Code
source('dependencies.R')
knitr::opts_chunk$set(echo = TRUE, warning=FALSE, message=FALSE)
#install.packages("stargazer")
library(stargazer)
#library(flexmix)
#library(qpcR)Theresa Szczepanski
December 7, 2023
The Massachusetts Education Reform Act was passed in 1993 in the context of a national movement toward education reform throughout the United States. As early as 1989 there were calls to establish national curriculum standards as a way to improve student college and career readiness skills and close poverty gaps (Greer 2018). Massachusetts Comprehensive Assessment System (MCAS) tests were introduced as part of the Massachusetts Education Reform Act.
The MCAS tests are a significant tool for educational equity. Scores on the Grade 10 Math MCAS test “predict longer-term educational attainments and labor market success, above and beyond typical markers of student advantage” and differences among students are largely and “sometimes completely accounted for” by differences in 10th grade MCAS scores and educational attainments. (Papay 2020).
With the introduction of the new Common Core standards and accountability testing came the demand for aligned curricular materials and teaching practices. Research indicates that the choice of instructional materials can have an impact “as large as or larger than the impact of teacher quality” (Chingos 2012). Massachusetts, along with Arkansas, Delaware, Kentucky, Louisiana, Maryland, Mississippi, Nebraska, New Mexico, Ohio, Rhode Island, Tennessee, and Texas belongs to the Council of Chief State School Officers’ (CCSO), High Quality Instructional Materials and Professional Development network which aims to close the “opportunity gap” among students by ensuring that every teacher has access to high-quality, standards aligned instructional materials and receives relevant professional development to support their use of these materials (Chief State School Officers 2021).
All Massachusetts Public School students must complete a high school science MCAS exam providing a wealth of standardized data on students’ discipline specific skill development. All schools receive annual summary reports on student performance. Significant work has been done using the MCAS achievement data and the Student Opportunity Act to identify achievement gaps and address funding inequities across the Commonwealth (Papay 2020). With funding gaps outlined in the late 1990’s closing, one could consider how the MCAS data could be leveraged to support the state’s current high quality instructional materials initiatives. The state compiles school’s performance disaggregated by each MCAS question item (DESE 2022).
Using the curricular information provided in statewide Next Generation MCAS High School Introductory Physics Item reports together with school-level student performance data, we hope to address the following broad questions:
Is there a relationship between differences in a school’s performance across Science Practice Categories and a school’s overall achievement on the Introductory Physics exam?
How can trends in a school’s performance be used to provide schools with guidance on discipline-specific curricular areas to target to improve student achievement?
In this report, I will analyze the High School Introductory Physics Next Generation Massachusetts Comprehensive Assessment System (MCAS) tests results for Massachusetts public schools.
Data for the study were drawn from DESE’s Next Generation MCAS Test Achievement Results statewide report, Item Analysis statewide report, and the MCAS digital item library. The Next Generation High School Introductory Physics MCAS assessment consists of 42 multiple choice and constructed response items that assess students on Physical Science standards from the 2016 STE Massachusetts Curriculum Framework in the content Reporting Categories of Motions and Forces, MF, Energy, EN, and Waves, WA. Each item is associated with a specific content standard from the Massachusetts Curriculum Framework as well as an underlying science Practice Category of Evidence Reasoning and Modeling, ERM, Mathematics and Data, MD, or Investigations and Questioning, IQ. The State Item Report provides the percentage of points earned by students in a school for each item as well as the percentage of points earned by all students in the state for each item.
The HSPhy_NextGen_SchoolSum data frame contains summary performance results from 112 public schools across the commonwealth on the Next Generation High School Introductory Physics MCAS, which was administered in the Spring of 2022 and 2023. 87 schools tested students in both years and 25 schools only tested students in 1 of the 2 testing years, with 27,745 students completing the exam.
For each school, there are values reported for 44 different variables which consist of information from three broad categories
School Characteristics: This includes the name of the school and the size of the school, School Size, as determined by the number of students that completed the MCAS exam.
Discipline-Specific Performance Metrics: This includes the percentage of points earned by students at a school for items in each content Reporting Category, MF%, EN%, WA% and science Practice Category ERM%, MD%, IQ%, the difference between a school’s percentage of points earned compared to the percentage of points earned by all students in the state (MFDiff, ENDiff, etc…), and the variability in a school’s performance relative to the state by category as measured by the standard deviation of the school’s Diff across categories (SD MF Diff, SD EN Diff, etc…).
Aggregate Performance Level metrics: This includes a school’s percentage of students at each of the four Performance Levels, (E%: Exceeding Expectations, M%: Meeting Expectations, PM%: Partially Meeting Expectations, and NM%: Not Meeting Expectations), the difference between these percentages and the percentage of students in Massachusetts at each performance level (EDiff, MDiff, PMDiff, NMDiff), and an ordinal classification of schools, EM Perf Stat, based on the percentage of students that were classified as Exceeding or Meeting expectations on the exam (HighEM, HighM, Mid, Mid-Low, Low).
See the HSPhy_NextGenMCASDF data frame summary and codebook for further details about all variables.
A school’s percentage of students classified as Exceeding or Meeting expectations on the Introductory Physics MCAS is negatively associated with a school’s variance in performance relative to students in the state on Mathematics and Data items, SD MD Diff.
A school’s summary performance on items in a given content Reporting Category as measured by MF%, EN%, and WA%, is positively associated with the Reporting Category's weight within the exam.
#HSPhy_NextGen_SchoolSum
HSPhy_NextGen_SchoolSum<-HSPhy_NextGen_SchoolSum%>%
ungroup()
#HSPhy_NextGen_SchoolSum
# HSPhy_NextGen_PerfDF
# HSPhy_NextGen_SchoolIT301DF
HSPhy_2023_SchoolSizeDF<-read_excel("data/2023_Physics_NextGenMCASItem.xlsx", skip = 1)%>%
select(`School Name`, `School Code`, `Tested`)%>%
mutate(`Tested` = as.integer(`Tested`))%>%
select(`School Name`, `School Code`, `Tested`)
HSPhy_2022_SchoolSizeDF<-read_excel("data/2022_Physics_NextGenMCASItem.xlsx", skip = 1)%>%
select(`School Name`, `School Code`, `Tested`)%>%
mutate(`Tested` = as.integer(`Tested`))%>%
select(`School Name`, `School Code`, `Tested`)
HSPhy_SchoolSize <- rbind(HSPhy_2023_SchoolSizeDF, HSPhy_2022_SchoolSizeDF)%>%
mutate(count = 1)%>%
group_by(`School Name`, `School Code`)%>%
summarise(count = sum(count),
`Tested` = sum(`Tested`))%>%
mutate(`Tested Count` = round(`Tested`/count))%>%
ungroup()
#HSPhy_SchoolSize
quantile <- quantile(HSPhy_SchoolSize$`Tested Count`)
HSPhy_Size<-HSPhy_SchoolSize%>%
mutate(`School Size` = case_when(
`Tested Count` <= quantile[2] ~ 1, # small
`Tested Count` > quantile[2] &
`Tested Count` <= quantile[3] ~ 2, #Low-Mid
`Tested Count` > quantile[3] &
`Tested Count` <= quantile[4] ~ 3, #Upper Mid
`Tested Count` > quantile[4] &
`Tested Count` <= quantile[5] ~ 4, # Large
))%>%
# mutate(`School Size` = recode_factor(`School Size`,
# "Small" = "Small",
# "Low-Mid" = "Low-Mid",
# "Upper-Mid" = "Upper-Mid",
# "Large" = "Large",
# .ordered = TRUE))%>%
select(`School Name`, `School Code`, `School Size`)
#HSPhy_Size
HSPhy_NextGen_SchoolSum<-HSPhy_NextGen_SchoolSum%>%
left_join(HSPhy_Size, by = c("School Name" = "School Name", "School Code" = "School Code"))%>%
mutate(`EMDiff` = `EDiff` + `MDiff`)%>%
mutate(`EM Perf Stat` = case_when(
`EDiff` > 0 & `EDiff` + `MDiff` > 0 ~ "HighEM",
`EDiff` <= 0 & `EDiff` + `MDiff` > 0 ~ "HighM",
#`EMDiff` > quantile(HSPhy_NextGen_SchoolSum$`EMDiff`)[3] &
`EMDiff` <= 0 & `EMDiff` > -14 ~ "Mid",
`EMDiff` <= -14 & `EMDiff` >= -33 ~ "Mid-Low",
`EMDiff` < -33 ~ "Low"
))%>%
mutate(`EM Perf Stat` = recode_factor(`EM Perf Stat`,
"HighEM" = "HighEM",
"HighM" = "HighM",
"Mid" = "Mid",
"Mid-Low" = "Mid-Low",
"Low" = "Low",
.ordered = TRUE))
HSPhy_NextGen_SchoolSum| Variable | Stats / Values | Freqs (% of Valid) | Graph | Missing | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Subject [character] | 1. PHY |
|
0 (0.0%) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| School Name [character] |
|
|
0 (0.0%) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| School Code [character] |
|
|
0 (0.0%) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| EN% [numeric] |
|
50 distinct values | 0 (0.0%) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| MF% [numeric] |
|
48 distinct values | 0 (0.0%) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| WA% [numeric] |
|
47 distinct values | 0 (0.0%) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| EN Diff SD [numeric] |
|
108 distinct values | 0 (0.0%) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| MF Diff SD [numeric] |
|
106 distinct values | 0 (0.0%) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| WA Diff SD [numeric] |
|
106 distinct values | 0 (0.0%) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| IQ% [numeric] |
|
55 distinct values | 0 (0.0%) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| MD% [numeric] |
|
48 distinct values | 0 (0.0%) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ERM% [numeric] |
|
49 distinct values | 0 (0.0%) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| None% [numeric] |
|
48 distinct values | 0 (0.0%) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| IQ Diff SD [numeric] |
|
66 distinct values | 10 (8.9%) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| MD Diff SD [numeric] |
|
107 distinct values | 0 (0.0%) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ERM Diff SD [numeric] |
|
101 distinct values | 0 (0.0%) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| None Diff SD [numeric] |
|
107 distinct values | 0 (0.0%) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Tested Students [integer] |
|
95 distinct values | 0 (0.0%) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| E% [numeric] |
|
29 distinct values | 0 (0.0%) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| M% [numeric] |
|
50 distinct values | 0 (0.0%) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| PM% [numeric] |
|
53 distinct values | 0 (0.0%) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| NM% [numeric] |
|
44 distinct values | 0 (0.0%) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| E%State [numeric] | 1 distinct value |
|
0 (0.0%) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| M%State [numeric] | 1 distinct value |
|
0 (0.0%) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| PM%State [numeric] | 1 distinct value |
|
0 (0.0%) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| NM%State [numeric] | 1 distinct value |
|
0 (0.0%) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| EDiff [numeric] |
|
29 distinct values | 0 (0.0%) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| MDiff [numeric] |
|
50 distinct values | 0 (0.0%) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| PMDiff [numeric] |
|
53 distinct values | 0 (0.0%) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| NMDiff [numeric] |
|
44 distinct values | 0 (0.0%) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| EN%State [numeric] | 1 distinct value |
|
0 (0.0%) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| MF%State [numeric] | 1 distinct value |
|
0 (0.0%) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| WA%State [numeric] | 1 distinct value |
|
0 (0.0%) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| IQ%State [numeric] | 1 distinct value |
|
0 (0.0%) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| MD%State [numeric] | 1 distinct value |
|
0 (0.0%) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ERM%State [numeric] | 1 distinct value |
|
0 (0.0%) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| None%State [numeric] | 1 distinct value |
|
0 (0.0%) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| School Size [numeric] |
|
|
0 (0.0%) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| EMDiff [numeric] |
|
64 distinct values | 0 (0.0%) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| EM Perf Stat [ordered, factor] |
|
|
0 (0.0%) |
Generated by summarytools 1.0.1 (R version 4.2.2)
2023-12-04
To explore the relationship between the distribution of a school’s students’ Performance Level and a school’s performance in content categories, we examine the percentage of points earned by students at schools as well as the standard deviation of the difference between points earned by students at a school and points earned by students in the state across Reporting Categories and Practice Categories. We grouped schools by their EM Perf Stat, an ordinal variable classifying schools by the percentage of students they have that were classified as either Exceeding or Meeting expectations on the MCAS. It appears as though items classified with the Science Practice Category of Mathematics and Data seem to be more challenging to students than those classified as Evidence, Reasoning, and Modeling. These practice categories are strongly and equally emphasized within the exam; items tagged with these categories account for 82% of the available points on the exam with exactly 41% of available points coming from each category.
When considering content Reporting Categories, there do not seem to be discernible distinctions between EM Perf Stat and school’s achievement and performance across categories. All schools appear to perform the strongest on Motion and Forces items, followed by Energy, and weakest on Waves items. Notably, this is also the order of the relative weights of the content areas within the exam; MF, EN, and WA items account for 50%, 30%, and 20% of exam points respectively.
When examining the statewide performance distribution, we can see from the right-skew that it is rare for schools to have high percentages of students classified as Not Meeting expectations and even rarer for schools to have high percentages of students classified as Exceeding expectations.
HSPhy_NextGen_SchoolSum%>%
select(`E%`, `M%`, `PM%`, `NM%`)%>%
pivot_longer(c(1:4), names_to = "Performance Level", values_to = "% Students")%>%
ggplot( aes(x=`% Students`, color=`Performance Level`, fill=`Performance Level`)) +
geom_histogram(alpha=0.6, binwidth = 15) +
scale_fill_viridis(discrete=TRUE) +
scale_color_viridis(discrete=TRUE) +
#theme_classic() +
theme(
legend.position="none",
panel.spacing = unit(0.1, "lines"),
strip.text.x = element_text(size = 8)
) +
facet_wrap(~`Performance Level`)+
labs( y = "",
title = "School Performance Level Distribution",
x = "% Students at Performance Level",
caption = "NextGen HS Physics MCAS") +scale_fill_brewer(palette="Paired") + scale_color_brewer(palette="Paired")Although Mathematics and Data and Evidence, Reasoning, and Modeling items have strong and equal weighting in the HS Introductory Physics exam, student performance distributions are noticeably different across these practice categories.
hist_practice_cat_achv<-HSPhy_NextGen_SchoolSum%>%
select(`ERM%`, `MD%`)%>%
pivot_longer(c(1:2), names_to = "Practice Cat", values_to = "% Points")%>%
ggplot( aes(x=`% Points`, color=`Practice Cat`, fill=`Practice Cat`)) +
geom_histogram(alpha=0.6, binwidth = 3) +
scale_fill_viridis(discrete=TRUE) +
scale_color_viridis(discrete=TRUE) +
#theme_ipsum() +
theme(
panel.spacing = unit(0.1, "lines"),
strip.text.x = element_text(size = 8)
) +
facet_wrap(~`Practice Cat`)+
labs( y = "",
title = "School Performance by Practice Category",
x = "% Points Earned",
caption = "NextGen HS Physics MCAS") +scale_fill_brewer(palette="Paired") + scale_color_brewer(palette="Paired")
#ggtitle("Practice Category Performance")
hist_practice_cat_achvWhen considering the variability of a school’s performance on items relative to the state by Practice Category, SD MD Diff, and SD ERM Diff, we can see that Mathematics and Data is skewed more to the right.
HSPhy_NextGen_SchoolSum%>%
select(`ERM Diff SD`, `MD Diff SD`)%>%
pivot_longer(c(1:2), names_to = "Practice Cat", values_to = "SD Diff")%>%
ggplot( aes(x=`SD Diff`, color=`Practice Cat`, fill=`Practice Cat`)) +
geom_histogram(alpha=0.6, binwidth = 3) +
scale_fill_viridis(discrete=TRUE) +
scale_color_viridis(discrete=TRUE) +
# theme_ipsum() +
theme(
panel.spacing = unit(0.1, "lines"),
strip.text.x = element_text(size = 8)
) +
labs( y = "",
title = "School Performance Variation by Practice Category",
x = "SD Diff",
caption = "NextGen HS Physics MCAS") +
facet_wrap(~`Practice Cat`) + scale_fill_brewer(palette="Paired") + scale_color_brewer(palette="Paired")These images seem to suggest that schools with the highest percentage of students classified as Exceeding or Meeting expectations on the MCAS have lower levels of variation in performance on Mathematics and Data Items and schools with the lowest percentage of students classified as Exceeding or Meeting expectations on the MCAS have the highest levels of variation in performance on Mathematics and Data Items.
box_practice_cat_diff<-HSPhy_NextGen_SchoolSum%>%
select(`EM Perf Stat`, `ERM Diff SD`, `MD Diff SD` )%>%
pivot_longer(c(2:3), names_to = "Practice Cat", values_to = "SD Diff")%>%
ggplot( aes(x= `EM Perf Stat`, y=`SD Diff`, fill= `EM Perf Stat`)) +
geom_boxplot() +
scale_fill_viridis(discrete = TRUE, alpha=0.6) +
geom_jitter(color="black", size=0.4, alpha=0.9) +
theme(
plot.title = element_text(size=11),
axis.title.x=element_blank(),
#axis.text.x=element_blank()
) +
labs( y = "SD Diff",
title = "Student Performance Variation by Practice Category",
x = "",
caption = "NextGen HS Physics MCAS") +
facet_wrap(~`Practice Cat`)+scale_fill_brewer(palette="Blues")
box_practice_cat_diffggsave("box_practice_cat_diff.png", device="png", width=8, height=6, units="in",dpi=300)
HSPhy_NextGen_SchoolSum%>%
select(`EM Perf Stat`, `ERM Diff SD`, `MD Diff SD` )%>%
pivot_longer(c(2:3), names_to = "Practice Cat", values_to = "SD Diff")%>%
ggplot( aes(x= `Practice Cat`, y=`SD Diff`, fill= `Practice Cat`)) +
geom_boxplot() +
scale_fill_viridis(discrete = TRUE, alpha=0.6) +
geom_jitter(color="black", size=0.4, alpha=0.9) +
#theme_ipsum() +
theme(
plot.title = element_text(size=11),
axis.title.x=element_blank(),
axis.text.x=element_blank()
) +
labs( y = "SD Diff",
title = "Student Practice Cat. Variation by Achievement Level",
x = "",
caption = "NextGen HS Physics MCAS") +
#xlab("")+
facet_wrap(~`EM Perf Stat`)+scale_fill_brewer(palette="Paired")These images seem to suggest that students at all schools seem to have more difficulty with Mathematics and Data items as compared to Evidence, Reasoning, and Modeling Items.
HSPhy_NextGen_SchoolSum%>%
select(`EM Perf Stat`, `ERM%`, `MD%` )%>%
pivot_longer(c(2:3), names_to = "Practice Cat", values_to = "%Points")%>%
ggplot( aes(x= `EM Perf Stat`, y=`%Points`, fill= `EM Perf Stat`)) +
geom_boxplot() +
scale_fill_viridis(discrete = TRUE, alpha=0.6) +
geom_jitter(color="black", size=0.4, alpha=0.9) +
#theme_ipsum() +
theme(
plot.title = element_text(size=11)
) +
labs( y = "%Points Earned",
title = "Student Practice Cat. Achievement by Performance Level",
x = "",
caption = "NextGen HS Physics MCAS") +
#xlab("")+
facet_wrap(~`Practice Cat`) +scale_fill_brewer(palette="Blues") box_practice_cat_achv<-HSPhy_NextGen_SchoolSum%>%
select(`EM Perf Stat`, `ERM%`, `MD%` )%>%
pivot_longer(c(2:3), names_to = "Practice Cat", values_to = "%Points")%>%
ggplot( aes(x= `Practice Cat`, y=`%Points`, fill= `Practice Cat`)) +
geom_boxplot() +
scale_fill_viridis(discrete = TRUE, alpha=0.6) +
geom_jitter(color="black", size=0.4, alpha=0.9) +
#theme_ipsum() +
theme(
plot.title = element_text(size=11)
) +
labs( y = "%Points Earned",
title = "Student Practice Cat. Achievement by Performance Level",
x = "",
caption = "NextGen HS Physics MCAS") +
#xlab("")+
facet_wrap(~`EM Perf Stat`, scale ="free_y") +scale_fill_brewer(palette="Paired")
box_practice_cat_achvHere we can visualize the variability of a school’s performance on items partitioned by Content Reporting Category of Motion and Forces, Energy, and Waves via: MF%/SD MF Diff, EN%/SD EN Diff, and WA%/SD WA Diff.
HSPhy_NextGen_SchoolSum%>%
select(`EM Perf Stat`, `MF Diff SD`, `EN Diff SD`, `WA Diff SD` )%>%
pivot_longer(c(2:4), names_to = "Report Cat", values_to = "SD Diff")%>%
ggplot( aes(x=`SD Diff`, color=`Report Cat`, fill=`Report Cat`, color = "grey")) +
geom_histogram(alpha=0.6, binwidth = 3) +
#scale_fill_viridis(discrete=TRUE) +
#scale_color_viridis(discrete=TRUE) +
#theme_ipsum() +
theme(
panel.spacing = unit(0.1, "lines"),
strip.text.x = element_text(size = 8)
) +
labs( y = "",
title = "School Performance Variation by Content Reporting Category",
x = "SD Diff",
caption = "NextGen HS Physics MCAS") +
facet_wrap(~`Report Cat`) + scale_fill_brewer(palette="Paired") + scale_color_brewer(palette="Paired") HSPhy_NextGen_SchoolSum%>%
select(`EM Perf Stat`, `MF%`, `EN%`, `WA%` )%>%
pivot_longer(c(2:4), names_to = "Report Cat", values_to = "% Points")%>%
ggplot( aes(x=`% Points`, color=`Report Cat`, fill=`Report Cat`)) +
geom_histogram(alpha=0.6, binwidth = 3) +
scale_fill_viridis(discrete=TRUE) +
scale_color_viridis(discrete=TRUE) +
#theme_ipsum() +
theme(
panel.spacing = unit(0.1, "lines"),
strip.text.x = element_text(size = 8)
) +
facet_wrap(~`Report Cat`)+
labs( y = "",
title = "Student Performance by Content Reporting Category",
x = "% Points Earned",
caption = "NextGen HS Physics MCAS") + scale_fill_brewer(palette="Paired") + scale_color_brewer(palette="Paired") These images suggest that schools seem to perform the best nn Motion and Forces items and the weakest on items in the Waves category. Most schools exhibit similar levels of variability in performance relative to the state across all reporting categories. Schools with the lowest percentage of students Exceeding expectations exhibit high variability in performance across all content reporting categories, but seem to have lower variability on Waves items.
box_report_cat_achv<-HSPhy_NextGen_SchoolSum%>%
select(`EM Perf Stat`, `MF%`, `EN%`, `WA%` )%>%
pivot_longer(c(2:4), names_to = "Report Cat", values_to = "% Points")%>%
ggplot( aes(x= `Report Cat`, y=`% Points`, fill= `Report Cat`)) +
geom_boxplot() +
scale_fill_discrete(breaks=c('MF%', 'EN%', 'WA%'))+
#scale_fill_viridis(discrete = TRUE, alpha=0.6) +
geom_jitter(color="black", size=0.4, alpha=0.9) +
#theme_ipsum() +
theme(
plot.title = element_text(size=11),
axis.title.x=element_blank(),
axis.text.x=element_blank()
) +
labs( y = "Report Cat%",
title = "School Content Reporting Cat. Performance by Achievement Level",
x = "",
caption = "NextGen HS Physics MCAS") +
#xlab("")+
facet_wrap(~`EM Perf Stat`, scales = "free") + scale_fill_brewer(palette="Blues")+
scale_x_discrete(limit = c("MF%", "EN%", "WA%"))
box_report_cat_achvggsave("box_report_cat_achv.png", device="png", width=8, height=6, units="in",dpi=300)
HSPhy_NextGen_SchoolSum%>%
select(`EM Perf Stat`, `MF Diff SD`, `EN Diff SD`, `WA Diff SD` )%>%
pivot_longer(c(2:4), names_to = "Report Cat", values_to = "SD Diff")%>%
ggplot( aes(x= `EM Perf Stat`, y=`SD Diff`, fill= `EM Perf Stat`)) +
geom_boxplot() +
scale_fill_viridis(discrete = TRUE, alpha=0.6) +
geom_jitter(color="black", size=0.4, alpha=0.9) +
theme(
plot.title = element_text(size=11),
axis.title.x=element_blank(),
axis.text.x=element_blank()
) +
labs( y = "SD Diff",
title = "School Performance Variation by Content Reporting Category",
x = "",
caption = "NextGen HS Physics MCAS") +
facet_wrap(~`Report Cat`) + scale_fill_brewer(palette="Blues") HSPhy_NextGen_SchoolSum%>%
select(`EM Perf Stat`, `MF Diff SD`, `EN Diff SD`, `WA Diff SD` )%>%
pivot_longer(c(2:4), names_to = "Report Cat", values_to = "SD Diff")%>%
ggplot( aes(x= `Report Cat`, y=`SD Diff`, fill= `Report Cat`)) +
geom_boxplot() +
scale_fill_viridis(discrete = TRUE, alpha=0.6) +
geom_jitter(color="black", size=0.4, alpha=0.9) +
#theme_ipsum() +
theme(
plot.title = element_text(size=11),
axis.title.x=element_blank(),
axis.text.x=element_blank()
) +
labs( y = "SD Diff",
title = "School Content Reporting Cat. Variation by Achievement Level",
x = "",
caption = "NextGen HS Physics MCAS") +
#xlab("")+
facet_wrap(~`EM Perf Stat`)+ scale_fill_brewer(palette="Blues") To explore the relationship between the variance in a school’s Diff compared to the state on Mathematics and Data items, MD Diff SD and a school’s percentage of students meeting or exceeding expectations on the MCAS, EorM%, we also will consider the impact of School Size, 1: Smallest, 4: Largest schools, as a control. It appears from our visuals that Small schools have a higher variation in mathematics and data items and smaller school’s typically perform worse on Mathematics and Data items and overall on the MCAS compared to larger schools.
HSPhy_NextGen_SchoolSum%>%
select(`School Size`, `MD Diff SD`, `ERM Diff SD` )%>%
pivot_longer(c(2:3), names_to = "Practice Cat", values_to = "SD Diff")%>%
ggplot( aes(x= `Practice Cat`, y=`SD Diff`, fill= `Practice Cat`)) +
geom_boxplot() +
scale_fill_viridis(discrete = TRUE, alpha=0.6) +
geom_jitter(color="black", size=0.4, alpha=0.9) +
#theme_ipsum() +
theme(
plot.title = element_text(size=11),
axis.title.x=element_blank(),
axis.text.x=element_blank()
) +
labs( y = "SD Diff",
title = "Student Practice Cat. Variation by School Size",
x = "",
caption = "NextGen HS Physics MCAS") +
#xlab("")+
facet_wrap(~`School Size`, scale = "free") + scale_fill_brewer(palette="Paired") HSPhy_NextGen_SchoolSum%>%
select(`School Size`, `MD%`, `ERM%` )%>%
pivot_longer(c(2:3), names_to = "Practice Cat", values_to = "%Points")%>%
ggplot( aes(x= `Practice Cat`, y=`%Points`, fill= `Practice Cat`)) +
geom_boxplot() +
scale_fill_viridis(discrete = TRUE, alpha=0.6) +
geom_jitter(color="black", size=0.4, alpha=0.9) +
#theme_ipsum() +
theme(
plot.title = element_text(size=11),
axis.title.x=element_blank(),
axis.text.x=element_blank()
) +
labs( y = "% Points Earned",
title = "Student Practice Cat. Achievement by School Size",
x = "",
caption = "NextGen HS Physics MCAS") +
#xlab("")+
facet_wrap(~`School Size`) + scale_fill_brewer(palette="Paired") However, when you group the schools by EM Perf Stat, you find that the highest performing, High EM, Small schools have a higher percentage of students meeting or exceeding expectations.
Across all sizes, it seems that the weakest performing schools have more variation in mathematics and data and the strongest performing schools have less variability in Mathematics and Data than in Evidence, Reasoning, and Modeling.
# HSPhy_NextGen_SchoolSum%>%
# select(`School Size`, `EorM%`, `MD Diff SD`, `EM Perf Stat` )%>%
# #pivot_longer(c(2:3), names_to = "Practice Cat", values_to = "SD Diff")%>%
# ggplot( aes(x= `School Size`, y=`EorM%`, fill= `School Size`)) +
# geom_boxplot() +
# scale_fill_viridis(discrete = TRUE, alpha=0.6) +
# geom_jitter(color="black", size=0.4, alpha=0.9) +
# #theme_ipsum() +
# theme(
#
# plot.title = element_text(size=11),
# axis.title.x=element_blank(),
# axis.text.x=element_blank()
# ) +
# labs( y = "SD Diff",
# title = "Students Meeting or Exceeding Expectations . Variation by Achievement Level",
# x = "",
# caption = "NextGen HS Physics MCAS") +
# #xlab("")+
# facet_wrap(~`EM Perf Stat`, scales = "free")Exceeding or Meeting expectations on the Introductory Physics MCAS is negatively associated with a school’s variance in performance relative to students in the state on Mathematics and Data items, SD MD Diff.I fit several models with EorM% as the response variable and MD Diff SD as the key explanatory variable.
I considered additional explanatory variables to control for the following:
Control for the variability in the other largest Science Practice Category, Evidence, Reasoning, and Modeling
ERM Diff SDControl for School Size
School SizeControl for the variability in the other content Reporting Categories: Waves, Motion and Forces, and Energy
WA Diff SDMF Diff SDEN Diff SDMD Diff SDWhen we consider MD Diff SD without controlling for other variables, our model associates a \(-3.9674\) drop in EorM% for a 1 point increase in MD Diff SD, \(p = 2.73e-09\).
Call:
lm(formula = `EorM%` ~ (`MD Diff SD`), data = HSPhy_NextGen_SchoolSum)
Residuals:
Min 1Q Median 3Q Max
-32.575 -18.347 -4.778 15.525 81.212
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 73.1813 5.6876 12.867 < 2e-16 ***
`MD Diff SD` -3.9674 0.6127 -6.476 2.73e-09 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 22.71 on 110 degrees of freedom
Multiple R-squared: 0.276, Adjusted R-squared: 0.2694
F-statistic: 41.94 on 1 and 110 DF, p-value: 2.726e-09
[1] 1021.306
[1] 1029.462
MD Diff SD + ERM Diff SDWhen we control for the variation in the other Science Practice Category, Evidence Reasoning, and Modeling, ERM Diff SD is not significant and does not have a significant interaction with MD Diff SD.
Call:
lm(formula = `EorM%` ~ (`MD Diff SD` + `ERM Diff SD`), data = HSPhy_NextGen_SchoolSum)
Residuals:
Min 1Q Median 3Q Max
-32.499 -18.077 -4.584 16.254 81.167
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 74.0590 6.7612 10.953 < 2e-16 ***
`MD Diff SD` -3.7413 1.1166 -3.351 0.00111 **
`ERM Diff SD` -0.3279 1.3517 -0.243 0.80876
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 22.81 on 109 degrees of freedom
Multiple R-squared: 0.2764, Adjusted R-squared: 0.2631
F-statistic: 20.82 on 2 and 109 DF, p-value: 2.201e-08
Call:
lm(formula = `EorM%` ~ (`MD Diff SD` + `ERM Diff SD` + `MD Diff SD` *
`ERM Diff SD`), data = HSPhy_NextGen_SchoolSum)
Residuals:
Min 1Q Median 3Q Max
-33.324 -17.081 -4.182 13.659 82.455
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 95.5577 16.4478 5.810 6.4e-08 ***
`MD Diff SD` -6.9179 2.4804 -2.789 0.00625 **
`ERM Diff SD` -2.1027 1.8288 -1.150 0.25279
`MD Diff SD`:`ERM Diff SD` 0.2558 0.1785 1.432 0.15491
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 22.7 on 108 degrees of freedom
Multiple R-squared: 0.2899, Adjusted R-squared: 0.2702
F-statistic: 14.7 on 3 and 108 DF, p-value: 4.302e-08
MD Diff SD + WA Diff SDWhen we control for the variation in the Content Reporting Category, Waves: WA Diff SD, without considering interaction, there is not a statistical significance to WA Diff SD.
However, when we include the interaction between WA Diff SD and MD Diff SD, we have that MD Diff SD is still statistically significant with an increase in MD Diff SD associated with a \(-10.02\) decrease in EorM%.
There is a statistically significant, \(p = 039\) interaction effect between WA Diff SD and MD Diff SD. For our model we have:
Call:
lm(formula = `EorM%` ~ (`MD Diff SD` + `WA Diff SD`), data = HSPhy_NextGen_SchoolSum)
Residuals:
Min 1Q Median 3Q Max
-31.508 -17.922 -6.025 14.653 74.863
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 67.3043 6.6291 10.153 < 2e-16 ***
`MD Diff SD` -5.1507 0.9278 -5.551 2.02e-07 ***
`WA Diff SD` 1.9300 1.1438 1.687 0.0944 .
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 22.52 on 109 degrees of freedom
Multiple R-squared: 0.2944, Adjusted R-squared: 0.2815
F-statistic: 22.74 on 2 and 109 DF, p-value: 5.561e-09
Call:
lm(formula = `EorM%` ~ (`MD Diff SD` + `WA Diff SD` + `MD Diff SD` *
`WA Diff SD`), data = HSPhy_NextGen_SchoolSum)
Residuals:
Min 1Q Median 3Q Max
-33.160 -16.523 -5.964 14.101 69.660
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 104.1191 18.7929 5.540 2.15e-07 ***
`MD Diff SD` -10.0194 2.5033 -4.002 0.000115 ***
`WA Diff SD` -1.7848 2.1050 -0.848 0.398373
`MD Diff SD`:`WA Diff SD` 0.4548 0.2177 2.089 0.039048 *
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 22.18 on 108 degrees of freedom
Multiple R-squared: 0.3218, Adjusted R-squared: 0.303
F-statistic: 17.09 on 3 and 108 DF, p-value: 3.762e-09
[1] 1017.981
[1] 1031.574
MD Diff SD, WA Diff SD, MD Diff SD*WA Diff SDWhen we consider adding a second content reporting category, Motion and Forces, only MD Diff SD is statistically significant. So we will control for content with just one reporting category, Waves.
Call:
lm(formula = `EorM%` ~ (`MD Diff SD` + `WA Diff SD` + `MD Diff SD` *
`WA Diff SD` + `MF Diff SD` + `MD Diff SD` * `MF Diff SD`),
data = HSPhy_NextGen_SchoolSum)
Residuals:
Min 1Q Median 3Q Max
-31.593 -16.746 -6.218 12.860 73.201
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 100.9689 19.1385 5.276 7.05e-07 ***
`MD Diff SD` -8.3768 3.4313 -2.441 0.0163 *
`WA Diff SD` 2.0642 3.1942 0.646 0.5195
`MF Diff SD` -4.2451 2.5290 -1.679 0.0962 .
`MD Diff SD`:`WA Diff SD` 0.1080 0.3061 0.353 0.7249
`MD Diff SD`:`MF Diff SD` 0.2522 0.1964 1.284 0.2019
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 22.09 on 106 degrees of freedom
Multiple R-squared: 0.3397, Adjusted R-squared: 0.3086
F-statistic: 10.91 on 5 and 106 DF, p-value: 1.754e-08
MD Diff SD + WA Diff SD, School SizeWhen we add the explanatory variable School Size to our model, WA Diff SD no longer has a statistically significant interaction with MD Diff SD. The only statistical significant explanatory variable is MD Diff SD.
MD Diff SD is associated with a \(-9.5141\) decrease in EorM%, \(p = .0008\)
Call:
lm(formula = `EorM%` ~ (`MD Diff SD` + `WA Diff SD` + `MD Diff SD` *
`WA Diff SD` + `School Size`), data = HSPhy_NextGen_SchoolSum)
Residuals:
Min 1Q Median 3Q Max
-34.140 -17.120 -6.408 14.187 70.851
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 94.8719 28.1816 3.366 0.001059 **
`MD Diff SD` -9.5141 2.7609 -3.446 0.000814 ***
`WA Diff SD` -1.2777 2.4047 -0.531 0.596279
`School Size` 1.3307 3.0130 0.442 0.659627
`MD Diff SD`:`WA Diff SD` 0.4215 0.2311 1.824 0.070985 .
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 22.26 on 107 degrees of freedom
Multiple R-squared: 0.3231, Adjusted R-squared: 0.2978
F-statistic: 12.77 on 4 and 107 DF, p-value: 1.57e-08
MD Diff SD School SizeWhen we control for the variation in School Size and remove content reporting categories from our model, we have that
A 1 point increase in MD Diff SD is associated with a \(-3.79\) decrease in EorM%, \(p = .0008\)
School Size is not statistically significant
\(p-value: 1.41e-09\)
adjusted \(r^2 = .26\)
AIC = 1023
BIC = 1034
Call:
lm(formula = `EorM%` ~ (`MD Diff SD` + `School Size`), data = HSPhy_NextGen_SchoolSum)
Residuals:
Min 1Q Median 3Q Max
-33.313 -18.006 -5.197 15.862 81.502
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 69.6836 13.0112 5.356 4.78e-07 ***
`MD Diff SD` -3.7916 0.8507 -4.457 2.03e-05 ***
`School Size` 0.7973 2.6650 0.299 0.765
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 22.8 on 109 degrees of freedom
Multiple R-squared: 0.2766, Adjusted R-squared: 0.2633
F-statistic: 20.84 on 2 and 109 DF, p-value: 2.168e-08
[1] 1023.214
[1] 1034.088
Since our original model, fit_md, has the 2nd lowest AIC, lowest BIC, and highest adjusted \(r^2\), and all other models had MD Diff SD with negative coefficients and having statistical significance, even when including controls, we can conclude
A school’s percentage of students classified as Exceeding or Meeting expectations on the Introductory Physics MCAS is negatively associated with a school’s variance in performance relative to students in the state on Mathematics and Data items, SD MD Diff.
EorM% = -3.67 * MD Diff SD + 40.330
High School Physics MCAS
=================================================================================================
Dependent variable:
-----------------------------------------------------------------------
EorM%
(1) (2) (3)
-------------------------------------------------------------------------------------------------
MD SD Diff -3.967*** -5.151*** -9.514***
(0.613) (0.928) (2.761)
WA SD Diff 1.930* -1.278
(1.144) (2.405)
School Size 1.331
(3.013)
`MD Diff SD`:`WA Diff SD` 0.422*
(0.231)
Constant 73.181*** 67.304*** 94.872***
(5.688) (6.629) (28.182)
-------------------------------------------------------------------------------------------------
Observations 112 112 112
R2 0.276 0.294 0.323
Adjusted R2 0.269 0.281 0.298
Residual Std. Error 22.707 (df = 110) 22.519 (df = 109) 22.262 (df = 107)
F Statistic 41.935*** (df = 1; 110) 22.743*** (df = 2; 109) 12.767*** (df = 4; 107)
=================================================================================================
Note: *p<0.1; **p<0.05; ***p<0.01
[1] 1023.214
[1] 1034.088
When we consider our diagnostics, our model seems to satisfy the assumptions of:
Linearity: our residuals plot appears to have a few outliers and a slightly negative slope
Normality of Errors: our standardized residuals seem to fall on the QQ line.
Homoskedasticity: there is a slight increase in our Scale-Location plot, but not very steep.
Our model does have a few terms that violate Cook’s distance. We have 112 and Cook’s distance says the 30th, 33rd, and 104th observations are in violation, since \(\frac{4}{n} \approx .04\).
Furthermore, when we remove those observations, and fit a new model, we find MD Diff SD continues to be statistically significant with a 1 point increase in MD Diff SD: \(p =6.62e-12\) associated with \(-4.8152\) decrease in EorM%.
For our adjusted model we have:
When we remove the outliers, the direction of our model and significance of our explanatory variables does not change.
[1] 0.03571429
Call:
lm(formula = `EorM%` ~ (`MD Diff SD`), data = test)
Residuals:
Min 1Q Median 3Q Max
-33.234 -17.490 -4.063 12.763 69.316
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 78.8842 5.5936 14.103 < 2e-16 ***
`MD Diff SD` -4.8152 0.6239 -7.717 6.62e-12 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 20.75 on 107 degrees of freedom
Multiple R-squared: 0.3576, Adjusted R-squared: 0.3516
F-statistic: 59.56 on 1 and 107 DF, p-value: 6.618e-12
[1] 974.3943
[1] 982.4684
This image displays the association between MD Diff SD and EorM%. One can see that high variability in Mathematics and Data items is associated with lower achievement for schools. Larger schools tend to perform better on the exam compared to smaller schools and have less variation in their mathematics performance although there are some small schools that have very high EorM%.
scat_md_size<-ggplot(data = HSPhy_NextGen_SchoolSum, aes(x = `MD Diff SD`, y = `EorM%`, color = `School Size`)) +
geom_point() +
geom_smooth(method="lm", se=T)+
labs( y = "EorM%",
title = "Variance in Mathematics vs. Student Achievement",
x = "MD Diff SD",
caption = "NextGen HS Physics MCAS")
scat_md_sizeReporting Category as measured by MF%, EN%, and WA%, is positively associated with the Reporting Category's weight within the exam.Report_Cat_WeightWhen we consider Report_Cat_Weight without controlling for other variables, our model associates a \(.24\) point increase in Report Cat% for each 1 point increase in Report_Cat_Weight, \(p = 6.54e-05\).
HSPhy_NextGen_SchoolSum<-HSPhy_NextGen_SchoolSum%>%
mutate(`MF_Weight` = round(100*30/60))%>%
mutate(`EN_Weight` = round(100*18/60))%>%
mutate(`WA_Weight` = round(100*12/60))
#HSPhy_NextGen_SchoolSum
HSPhy_Weight<-HSPhy_NextGen_SchoolSum%>%
select(`MF_Weight`, `EN_Weight`, `WA_Weight`, `MD Diff SD`, `School Size`, `School Name`, `School Code`)%>%
pivot_longer(c(1:3), names_to = "Report Cat", values_to = "Exam Weigt") #%>%
#HSPhy_Weight
HSPhy_Cat<-HSPhy_NextGen_SchoolSum%>%
select(`MF%`, `EN%`, `WA%`, `School Code`, `MD Diff SD`, `School Size`)%>%
pivot_longer(c(1:3), names_to = "Report Cat", values_to = "Report Cat%")%>%
group_by
HSPhy_Cat<-HSPhy_Cat%>%
mutate(Report_Cat_Weight = case_when(
`Report Cat` == "MF%" ~ 50,
`Report Cat` == "EN%" ~ 30,
`Report Cat` == "WA%" ~ 20,
))
#HSPhy_Cat
# HSPhy_Weight<- HSPhy_Weight%>%
# select(`School Name`, `School Code`, `Report Cat`, `Exam Weight`, `% Points`)%>%
# group_by(`School Name`, `School Code`, `Report Cat`)%>%
# summarize(`Exam Weight` = mean(`Exam Weight`),
# `Report Cat %` = mean(`% Points`))%>%
# ungroup()
#
#HSPhy_Cat
fit_weight = lm(`Report Cat%` ~ `Report_Cat_Weight`, data = HSPhy_Cat)
summary(fit_weight)
Call:
lm(formula = `Report Cat%` ~ Report_Cat_Weight, data = HSPhy_Cat)
Residuals:
Min 1Q Median 3Q Max
-30.272 -9.496 -0.384 9.046 33.893
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 40.33036 2.10210 19.186 < 2e-16 ***
Report_Cat_Weight 0.23884 0.05906 4.044 6.54e-05 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 13.5 on 334 degrees of freedom
Multiple R-squared: 0.04667, Adjusted R-squared: 0.04382
F-statistic: 16.35 on 1 and 334 DF, p-value: 6.537e-05
[1] 2706.685
[1] 2718.136
Report_Cat_Weight, MD Diff SDWhen we control for the variation in the Science Practice Category, Mathematics and Data: MD Diff SD,
An increase in Report_Cat_Weight: \(1.05e-06\) is associated with a \(.24\) increase in Report Cat%
An increase in MD Diff SD: \(p < 2e-16\) is associated with a \(-2.25\) decrease in Report Cat%.
For our model we have:
Call:
lm(formula = `Report Cat%` ~ Report_Cat_Weight + `MD Diff SD`,
data = HSPhy_Cat)
Residuals:
Min 1Q Median 3Q Max
-18.480 -8.721 -2.223 6.604 37.986
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 59.63911 2.25386 26.461 < 2e-16 ***
Report_Cat_Weight 0.23884 0.04801 4.975 1.05e-06 ***
`MD Diff SD` -2.24586 0.17097 -13.136 < 2e-16 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 10.98 on 333 degrees of freedom
Multiple R-squared: 0.3721, Adjusted R-squared: 0.3683
F-statistic: 98.66 on 2 and 333 DF, p-value: < 2.2e-16
[1] 2568.397
[1] 2583.665
Report_Cat_Weight, MD Diff SD, with InteractionWhen we test for interaction between Reporting Category Weight: Report_Cat_Weight and Mathematics and Data: MD Diff SD,
Call:
lm(formula = `Report Cat%` ~ Report_Cat_Weight + `MD Diff SD` +
Report_Cat_Weight * `MD Diff SD`, data = HSPhy_Cat)
Residuals:
Min 1Q Median 3Q Max
-18.338 -8.617 -2.226 6.515 37.923
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 58.567562 4.535389 12.913 < 2e-16 ***
Report_Cat_Weight 0.270986 0.127433 2.126 0.0342 *
`MD Diff SD` -2.121222 0.488546 -4.342 1.88e-05 ***
Report_Cat_Weight:`MD Diff SD` -0.003739 0.013727 -0.272 0.7855
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 10.99 on 332 degrees of freedom
Multiple R-squared: 0.3722, Adjusted R-squared: 0.3665
F-statistic: 65.61 on 3 and 332 DF, p-value: < 2.2e-16
Report_Cat_Weight, MD Diff SD, School SizeWhen we control for the variation in MD Diff SD and School Size, we have that Report_Cat_Weight: \(p =4.37e-07\), MD Diff SD: \(p = 0.0318\), and School Size: \(p = 4.33e-06\) are statistically significant and there is a statistically significant, \(p = 4.74e-07\), interaction effect between MD Diff SD and School Size.
For our model we have:
an increase in Report_Cat_Weight is associated with a \(.24\) increase in Report Cat%
E(Report Cat%) = 48.29 + 6.83 * School Size - .77* MD Diff SD - .98(MD Diff SD * School Size)
\(p-value: 2.2e-16\)
adjusted \(r^2 = 0.41\)
AIC = 2546
BIC = 2569
When we include an interaction between School Size and MD Diff SD, there is a significant interaction, \(p = 4.74e-07\).
Call:
lm(formula = `Report Cat%` ~ Report_Cat_Weight + `MD Diff SD` +
`School Size` + `School Size` * `MD Diff SD`, data = HSPhy_Cat)
Residuals:
Min 1Q Median 3Q Max
-19.175 -7.607 -1.923 6.314 36.632
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 48.28948 4.29490 11.243 < 2e-16 ***
Report_Cat_Weight 0.23884 0.04633 5.155 4.37e-07 ***
`MD Diff SD` -0.77048 0.35745 -2.155 0.0318 *
`School Size` 6.83058 1.46172 4.673 4.33e-06 ***
`MD Diff SD`:`School Size` -0.98064 0.19083 -5.139 4.74e-07 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 10.59 on 331 degrees of freedom
Multiple R-squared: 0.4187, Adjusted R-squared: 0.4117
F-statistic: 59.61 on 4 and 331 DF, p-value: < 2.2e-16
[1] 2546.464
[1] 2569.367
==============================================================================================================================
Dependent variable:
-----------------------------------------------------------------------------------------------
Report Cat%
(1) (2) (3) (4)
------------------------------------------------------------------------------------------------------------------------------
Report_Cat_Weight 0.239*** 0.239*** 0.271** 0.239***
(0.059) (0.048) (0.127) (0.046)
`MD Diff SD` -2.246*** -2.121*** -0.770**
(0.171) (0.489) (0.357)
Report_Cat_Weight:`MD Diff SD` -0.004
(0.014)
`School Size` 6.831***
(1.462)
`MD Diff SD`:`School Size` -0.981***
(0.191)
Constant 40.330*** 59.639*** 58.568*** 48.289***
(2.102) (2.254) (4.535) (4.295)
------------------------------------------------------------------------------------------------------------------------------
Observations 336 336 336 336
R2 0.047 0.372 0.372 0.419
Adjusted R2 0.044 0.368 0.367 0.412
Residual Std. Error 13.503 (df = 334) 10.975 (df = 333) 10.991 (df = 332) 10.592 (df = 331)
F Statistic 16.352*** (df = 1; 334) 98.655*** (df = 2; 333) 65.612*** (df = 3; 332) 59.605*** (df = 4; 331)
==============================================================================================================================
Note: *p<0.1; **p<0.05; ***p<0.01
Since this model has the lowest AIC, lowest BIC, highest adjusted \(r^2\), and all terms are statistically significant, we can conclude
A school’s summary performance on items in a given content Reporting Category as measured by Report Cat%, is positively associated with the Reporting Category's weight within the exam. Report_Cat Weight was statistically significant, even when we control for impact of MD Diff SD andSchool Size.
Report Cat% = .239 * Report_Cat_Weight - .77 * MD Diff SD + 6.831 * School Size
It should be noted, that there was a very large increase from \(.04\), to \(.37\) in \(r^2\) once we added MD Diff SD as an explanatory variable to our model.
fit_weight_md_sizeWhen we consider our diagnostics, our model seems to satisfy the assumptions of:
Linearity: our residuals plot appears to have a few outliers with the remaining values forming a horizontal band around the 0 line.
Normality of Errors: Our standardized residual points generally fall along the QQ line.
Homoskedasticity: there is a slight increase in our Scale-Location plot, but not very steep.
Our model does have some terms that violate Cook’s distance. We have 336 observations and Cook’s distance says the 141st, 310th, and 311th observations are in violation, since \(\frac{4}{n} \approx .01\).
When we remove those observations, and fit a new model, we find all of our explanatory variables to continue to be statistically significant
For our adjusted model we have:
When we remove the outliers, the direction of our model, significance of our explanatory variables, and signs of their coefficients do not change.
[1] 0.01190476
Call:
lm(formula = `Report Cat%` ~ Report_Cat_Weight + `MD Diff SD` +
`School Size` + `School Size` * `MD Diff SD`, data = test)
Residuals:
Min 1Q Median 3Q Max
-18.819 -6.956 -1.755 6.434 25.277
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 46.86362 3.92096 11.952 < 2e-16 ***
Report_Cat_Weight 0.24110 0.04243 5.683 2.95e-08 ***
`MD Diff SD` -0.92949 0.32627 -2.849 0.00467 **
`School Size` 6.84980 1.32939 5.153 4.47e-07 ***
`MD Diff SD`:`School Size` -0.87165 0.17376 -5.016 8.68e-07 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 9.613 on 325 degrees of freedom
Multiple R-squared: 0.4926, Adjusted R-squared: 0.4863
F-statistic: 78.87 on 4 and 325 DF, p-value: < 2.2e-16
[1] 2437.103
[1] 2459.898
Report_Cat_Weight, MD Diff SD, School SizeHSPhy_Cat_Plot<-HSPhy_Cat%>%
mutate(`School Size Fact` = case_when(
`School Size` == 1 ~ "Small",
`School Size` == 2 ~ "Low-Mid",
`School Size` == 3 ~ "Upper-Mid",
`School Size` == 4 ~ "Large"
))%>%
mutate(`School Size Fact` = recode_factor(`School Size Fact`,
"Small" = "Small",
"Low-Mid" = "Low-Mid",
"Upper-Mid" = "Upper-Mid",
"Large" = "Large",
.ordered = TRUE))
#HSPhy_Cat_Plot
mod_plot1<-ggplot(data = HSPhy_Cat_Plot, aes(x = `MD Diff SD`, y = `Report Cat%`, color = `School Size Fact`))+
geom_point() +
geom_smooth(method="lm")+
labs( y = "Report Cat%",
title = "MD SD Diff and School Size vs. Achievement",
x = "MD Diff SD",
caption = "NextGen HS Physics MCAS")+scale_color_brewer(palette="Paired")
mod_plot1scat_md_weight<-ggplot(data = HSPhy_Cat, aes(x = `MD Diff SD`, y = `Report Cat%`, color = `Report_Cat_Weight`)) +
geom_point() +
geom_smooth(method="lm", se=T)+
labs( y = "Report Cat%",
title = "Variance in Mathematics and Category Weight vs. Achievement",
x = "MD Diff SD",
caption = "NextGen HS Physics MCAS")#+
#facet_wrap(~`Report_Cat_Weight`, scales = "free")
scat_md_weight# ggplot(data = HSPhy_Cat, aes(x = `MD Diff SD`, y = `Report Cat%`, color = `School Size`)) +
# geom_point() +
# geom_smooth(method="lm", se=T)+
# labs( y = "Report Cat%",
# title = "Variance in Mathematics and School Size vs. Achievement",
# x = "MD Diff SD",
# caption = "NextGen HS Physics MCAS")+
# facet_wrap(~`Report_Cat_Weight`, scales = "free")# ggplot(data = HSPhy_Cat, aes(x = `MD Diff SD`, y = `Report Cat%`, color = `Report_Cat_Weight`)) +
# geom_point() +
# geom_smooth(method="lm", se=T)+
# labs( y = "Report Cat%",
# title = "Variance in Mathematics and Category Weight vs. Achievement",
# x = "MD Diff SD",
# caption = "NextGen HS Physics MCAS")+
# facet_wrap(~`Report_Cat_Weight`, scales = "free")# ggplot(data = HSPhy_Cat, aes(x = `MD Diff SD`, y = `Report Cat%`, color = `School Size`)) +
# geom_point() +
# geom_smooth(method="lm")+
# labs( y = "Report Cat%",
# title = "MD SD Diff, Category Weight, and School Size vs. Achievement",
# x = "MD Diff SD",
# caption = "NextGen HS Physics MCAS")+
# facet_wrap(~`Report_Cat_Weight`, scales = "free")#mod_plot<-ggplot(data = HSPhy_Cat, aes(x = `MD Diff SD`, y = `Report Cat%`, color = `School Size`))
#mod_plot + geom_point() + geom_smooth(method = lm,se = FALSE, fullrange = TRUE) #+ #scale_color_brewer(type = "qual", palette = 3)
# geom_point() +
# geom_smooth(method="lm")+
# labs( y = "Report Cat%",
# title = "MD SD Diff, Category Weight, and School Size vs. Achievement",
# x = "MD Diff SD",
# caption = "NextGen HS Physics MCAS")+
# facet_wrap(~`Report_Cat_Weight`, scales = "free")When considering the performance of Massachusetts students on the Next Generation High School Introductory Physics Exam, we find that:
Reporting Cat% and EorM% were negatively associated with increased variability in Mathematics and Data, MD Diff SD.This suggests that schools could benefit by spending time reviewing and supporting science teachers in implementing evidence based, best practices for the teaching of Mathematics within the Physics classroom.
Reporting Cat% was positively associated with the Report_Cat_Weight.There was no significant interaction between Reporting Cat% and MD Diff SD. This suggests that schools could benefit by creating a scope and sequence for the calendar year that ensures that all content areas from the Massachusetts Curriculum framework are addressed within the academic year. Perhaps items in the Waves category are not receiving enough instructional time within the school year as compared to those in the highest weighted Motion and Forces category.
One could examine the G8 Science exam as well as the G5 Science exam by Science Practice Category and Content Reporting Category to see if there are similar trends exhibited in earlier grade levels.
---
title: "DACSS603Final"
author: "Theresa Szczepanski"
desription: "MCAS G9 Science Analysis"
date: "12/07/2023"
format:
html:
embed-resources: true
self-contained-math: true
df-print: paged
toc: true
code-fold: true
code-copy: true
code-tools: true
bibliography: references.bib
editor:
markdown:
wrap: 72
---
```{r}
#| label: setup
#| warning: false
#| message: false
source('dependencies.R')
knitr::opts_chunk$set(echo = TRUE, warning=FALSE, message=FALSE)
#install.packages("stargazer")
library(stargazer)
#library(flexmix)
#library(qpcR)
```
# Research Questions
The Massachusetts Education Reform Act was passed in 1993 in the context
of a national movement toward education reform throughout the United
States. As early as 1989 there were calls to establish national
curriculum standards as a way to improve student college and career
readiness skills and close poverty gaps [@Greer18]. Massachusetts
Comprehensive Assessment System (MCAS) tests were introduced as part of
the Massachusetts Education Reform Act.
The MCAS tests are a significant tool for educational equity. Scores on
the Grade 10 Math MCAS test "predict longer-term educational attainments
and labor market success, above and beyond typical markers of student
advantage" and differences among students are largely and "sometimes
completely accounted for" by differences in 10th grade MCAS scores and
educational attainments. [@Boats20].
With the introduction of the new Common Core standards and
accountability testing came the demand for aligned curricular materials
and teaching practices. Research indicates that the choice of
instructional materials can have an impact "as large as or larger than
the impact of teacher quality" [@Blindly12]. Massachusetts, along with
Arkansas, Delaware, Kentucky, Louisiana, Maryland, Mississippi,
Nebraska, New Mexico, Ohio, Rhode Island, Tennessee, and Texas belongs
to the Council of Chief State School Officers' (CCSO), [High Quality
Instructional Materials and Professional Development
network](https://learning.ccsso.org/high-quality-instructional-materials)
which aims to close the "opportunity gap" among students by ensuring
that every teacher has access to high-quality, standards aligned
instructional materials and receives relevant professional development
to support their use of these materials [@IMPD21].
All Massachusetts Public School students must complete a high school
science MCAS exam providing a wealth of standardized data on students'
discipline specific skill development. All schools receive annual
summary reports on student performance. Significant work has been done
using the MCAS achievement data and the Student Opportunity Act to
identify achievement gaps and address funding inequities across the
Commonwealth [@Boats20]. With funding gaps outlined in the late 1990's
closing, one could consider how the MCAS data could be leveraged to
support the state's current high quality instructional materials
initiatives. The state compiles school's performance disaggregated by
each MCAS question item [@MCASIT].
Using the curricular information provided in statewide Next Generation
MCAS High School Introductory Physics Item reports together with
school-level student performance data, we hope to address the following
broad questions:
```{=html}
<style>
div.blue { background-color:#e6f0ff; border-radius: 5px; padding: 20px;}
</style>
```
::: blue
- Is there a relationship between differences in a school's
performance across Science Practice Categories and a school's
overall achievement on the Introductory Physics exam?
- How can trends in a school's performance be used to provide schools
with guidance on discipline-specific curricular areas to target to
improve student achievement?
:::
In this report, I will analyze the High School Introductory Physics Next
Generation [Massachusetts Comprehensive Assessment System
(MCAS)](https://www.doe.mass.edu/mcas/default.html) tests results for
Massachusetts public schools.
Data for the study were drawn from DESE's Next Generation MCAS Test
[Achievement Results statewide
report](https://profiles.doe.mass.edu/statereport/mcas.aspx), [Item
Analysis statewide
report](https://profiles.doe.mass.edu/statereport/nextgenmcas_item.aspx),
and the [MCAS digital item
library](https://mcas.digitalitemlibrary.com/home?subject=Science&grades=Physics&view=ALL).
The Next Generation High School Introductory Physics MCAS assessment
consists of 42 multiple choice and constructed response items that
assess students on Physical Science standards from the [2016 STE
Massachusetts Curriculum
Framework](https://www.doe.mass.edu/frameworks/scitech/2016-04.pdf) in
the content `Reporting Categories` of Motions and Forces, `MF`, Energy,
`EN`, and Waves, `WA`. Each item is associated with a specific content
standard from the Massachusetts Curriculum Framework as well as an
underlying science `Practice Category` of Evidence Reasoning and
Modeling, `ERM`, Mathematics and Data, `MD`, or Investigations and
Questioning, `IQ`. The State Item Report provides the percentage of
points earned by students in a school for each item as well as the
percentage of points earned by all students in the state for each item.
The `HSPhy_NextGen_SchoolSum` data frame contains summary performance
results from 112 public schools across the commonwealth on the Next
Generation High School Introductory Physics MCAS, which was administered
in the Spring of 2022 and 2023. 87 schools tested students in both years
and 25 schools only tested students in 1 of the 2 testing years, with
27,745 students completing the exam.
For each school, there are values reported for 44 different variables
which consist of information from three broad categories
- *School Characteristics*: This includes the name of the school and
the size of the school, `School Size`, as determined by the number
of students that completed the MCAS exam.
- *Discipline-Specific Performance Metrics*: This includes the
percentage of points earned by students at a school for items in each
content `Reporting Category`, `MF%`, `EN%`, `WA%` and science
`Practice Category` `ERM%`, `MD%`, `IQ%`, the difference between a
school's percentage of points earned compared to the percentage of
points earned by all students in the state (`MFDiff`, `ENDiff`,
etc...), and the variability in a school's performance relative to
the state by category as measured by the standard deviation of the
school's `Diff` across categories (`SD MF Diff`, `SD EN Diff`,
etc...).
- *Aggregate Performance Level metrics*: This includes a school's
percentage of students at each of the four `Performance Levels`,
(`E%`: Exceeding Expectations, `M%`: Meeting Expectations, `PM%`:
Partially Meeting Expectations, and `NM%`: Not Meeting
Expectations), the difference between these percentages and the
percentage of students in Massachusetts at each performance level
(`EDiff`, `MDiff`, `PMDiff`, `NMDiff`), and an ordinal
classification of schools, `EM Perf Stat`, based on the percentage
of students that were classified as Exceeding or Meeting
expectations on the exam (`HighEM`, `HighM`, `Mid`, `Mid-Low`,
`Low`).
See the `HSPhy_NextGenMCASDF` data frame summary and **codebook** for
further details about all variables.
# Hypotheses
```{=html}
<style>
div.blue { background-color:#e6f0ff; border-radius: 5px; padding: 20px;}
</style>
```
::: blue
- A school's percentage of students classified as `Exceeding` or `Meeting`
expectations on the Introductory Physics MCAS is negatively
associated with a school's variance in performance relative to
students in the state on `Mathematics and Data` items, `SD MD Diff`.
- A school's summary performance on items in a given content
`Reporting Category` as measured by `MF%`, `EN%`, and `WA%`, is
positively associated with the `Reporting Category's` weight within
the exam.
:::
# Descriptive Statistics
```{r}
#| label: dataframe setup
#| warning: false
#| message: false
#HSPhy_NextGen_SchoolSum
HSPhy_NextGen_SchoolSum<-HSPhy_NextGen_SchoolSum%>%
ungroup()
#HSPhy_NextGen_SchoolSum
# HSPhy_NextGen_PerfDF
# HSPhy_NextGen_SchoolIT301DF
HSPhy_2023_SchoolSizeDF<-read_excel("data/2023_Physics_NextGenMCASItem.xlsx", skip = 1)%>%
select(`School Name`, `School Code`, `Tested`)%>%
mutate(`Tested` = as.integer(`Tested`))%>%
select(`School Name`, `School Code`, `Tested`)
HSPhy_2022_SchoolSizeDF<-read_excel("data/2022_Physics_NextGenMCASItem.xlsx", skip = 1)%>%
select(`School Name`, `School Code`, `Tested`)%>%
mutate(`Tested` = as.integer(`Tested`))%>%
select(`School Name`, `School Code`, `Tested`)
HSPhy_SchoolSize <- rbind(HSPhy_2023_SchoolSizeDF, HSPhy_2022_SchoolSizeDF)%>%
mutate(count = 1)%>%
group_by(`School Name`, `School Code`)%>%
summarise(count = sum(count),
`Tested` = sum(`Tested`))%>%
mutate(`Tested Count` = round(`Tested`/count))%>%
ungroup()
#HSPhy_SchoolSize
quantile <- quantile(HSPhy_SchoolSize$`Tested Count`)
HSPhy_Size<-HSPhy_SchoolSize%>%
mutate(`School Size` = case_when(
`Tested Count` <= quantile[2] ~ 1, # small
`Tested Count` > quantile[2] &
`Tested Count` <= quantile[3] ~ 2, #Low-Mid
`Tested Count` > quantile[3] &
`Tested Count` <= quantile[4] ~ 3, #Upper Mid
`Tested Count` > quantile[4] &
`Tested Count` <= quantile[5] ~ 4, # Large
))%>%
# mutate(`School Size` = recode_factor(`School Size`,
# "Small" = "Small",
# "Low-Mid" = "Low-Mid",
# "Upper-Mid" = "Upper-Mid",
# "Large" = "Large",
# .ordered = TRUE))%>%
select(`School Name`, `School Code`, `School Size`)
#HSPhy_Size
HSPhy_NextGen_SchoolSum<-HSPhy_NextGen_SchoolSum%>%
left_join(HSPhy_Size, by = c("School Name" = "School Name", "School Code" = "School Code"))%>%
mutate(`EMDiff` = `EDiff` + `MDiff`)%>%
mutate(`EM Perf Stat` = case_when(
`EDiff` > 0 & `EDiff` + `MDiff` > 0 ~ "HighEM",
`EDiff` <= 0 & `EDiff` + `MDiff` > 0 ~ "HighM",
#`EMDiff` > quantile(HSPhy_NextGen_SchoolSum$`EMDiff`)[3] &
`EMDiff` <= 0 & `EMDiff` > -14 ~ "Mid",
`EMDiff` <= -14 & `EMDiff` >= -33 ~ "Mid-Low",
`EMDiff` < -33 ~ "Low"
))%>%
mutate(`EM Perf Stat` = recode_factor(`EM Perf Stat`,
"HighEM" = "HighEM",
"HighM" = "HighM",
"Mid" = "Mid",
"Mid-Low" = "Mid-Low",
"Low" = "Low",
.ordered = TRUE))
HSPhy_NextGen_SchoolSum
#quantile(HSPhy_NextGen_SchoolSum$`EMDiff`)
#summary(HSPhy_NextGen_SchoolSum)
print(summarytools::dfSummary(HSPhy_NextGen_SchoolSum,
varnumbers = FALSE,
plain.ascii = FALSE,
style = "grid",
graph.magnif = 0.70,
valid.col = FALSE),
method = 'render',
table.classes = 'table-condensed')
```
## Key Variables
To explore the relationship between the distribution of a school's
students' `Performance Level` and a school's performance in content
categories, we examine the percentage of points earned by students at
schools as well as the standard deviation of the difference between
points earned by students at a school and points earned by students in
the state across `Reporting Categories` and `Practice Categories`. We
grouped schools by their `EM Perf Stat`, an ordinal variable classifying
schools by the percentage of students they have that were classified as
either Exceeding or Meeting expectations on the MCAS. It appears as though items
classified with the Science `Practice Category` of
`Mathematics and Data` seem to be more challenging to students than
those classified as `Evidence, Reasoning, and Modeling`. These practice
categories are strongly and equally emphasized within the exam; items
tagged with these categories account for **82%** of the available points
on the exam with exactly **41%** of available points coming from each
category.
When considering content `Reporting Categories`, there do not seem to be
discernible distinctions between `EM Perf Stat` and school's achievement
and performance across categories. All schools appear to perform the
strongest on `Motion and Forces` items, followed by `Energy`, and
weakest on `Waves` items. Notably, this is also the order of the
relative weights of the content areas within the exam; `MF`, `EN`, and
`WA` items account for **50%**, **30%**, and **20%** of exam points
respectively.
```{r}
#quantile(HSPhy_NextGen_SchoolSum$`EMDiff`)
HSPhy_NextGen_SchoolSum%>%
group_by(`EM Perf Stat`)%>%
summarise( `Mean MD%` = mean(`MD%`),
`Mean MD SD` = mean(`MD Diff SD`),
`Mean ERM%` = mean(`ERM%`),
`Mean ERM SD` = mean(`ERM Diff SD`))
HSPhy_NextGen_SchoolSum%>%
group_by(`EM Perf Stat`)%>%
summarise( `Mean MF%` = mean(`MF%`),
`Mean MF SD` = mean(`MF Diff SD`),
`Mean EN%` = mean(`EN%`),
`Mean EN SD` = mean (`EN Diff SD`),
`Mean WA%` = mean(`WA%`),
`Mean WA SD` = mean (`WA Diff SD`)
)
```
# Visualization
## Distribution of Performance Level %
When examining the statewide performance distribution, we can see from
the right-skew that it is rare for schools to have high percentages of
students classified as `Not Meeting` expectations and even rarer for
schools to have high percentages of students classified as `Exceeding`
expectations.
```{r}
HSPhy_NextGen_SchoolSum%>%
select(`E%`, `M%`, `PM%`, `NM%`)%>%
pivot_longer(c(1:4), names_to = "Performance Level", values_to = "% Students")%>%
ggplot( aes(x=`% Students`, color=`Performance Level`, fill=`Performance Level`)) +
geom_histogram(alpha=0.6, binwidth = 15) +
scale_fill_viridis(discrete=TRUE) +
scale_color_viridis(discrete=TRUE) +
#theme_classic() +
theme(
legend.position="none",
panel.spacing = unit(0.1, "lines"),
strip.text.x = element_text(size = 8)
) +
facet_wrap(~`Performance Level`)+
labs( y = "",
title = "School Performance Level Distribution",
x = "% Students at Performance Level",
caption = "NextGen HS Physics MCAS") +scale_fill_brewer(palette="Paired") + scale_color_brewer(palette="Paired")
```
## Distribution of School Performance and Variability by Practice Cat
Although `Mathematics and Data` and `Evidence, Reasoning, and Modeling`
items have strong and equal weighting in the HS Introductory Physics
exam, student performance distributions are noticeably different across
these practice categories.
```{r}
hist_practice_cat_achv<-HSPhy_NextGen_SchoolSum%>%
select(`ERM%`, `MD%`)%>%
pivot_longer(c(1:2), names_to = "Practice Cat", values_to = "% Points")%>%
ggplot( aes(x=`% Points`, color=`Practice Cat`, fill=`Practice Cat`)) +
geom_histogram(alpha=0.6, binwidth = 3) +
scale_fill_viridis(discrete=TRUE) +
scale_color_viridis(discrete=TRUE) +
#theme_ipsum() +
theme(
panel.spacing = unit(0.1, "lines"),
strip.text.x = element_text(size = 8)
) +
facet_wrap(~`Practice Cat`)+
labs( y = "",
title = "School Performance by Practice Category",
x = "% Points Earned",
caption = "NextGen HS Physics MCAS") +scale_fill_brewer(palette="Paired") + scale_color_brewer(palette="Paired")
#ggtitle("Practice Category Performance")
hist_practice_cat_achv
ggsave("hist_practice_cat_achv.png", device="png", width=8, height=4, units="in",dpi=300)
```
When considering the variability of a school's performance on items
relative to the state by `Practice Category`, `SD MD Diff`, and
`SD ERM Diff`, we can see that `Mathematics and Data` is skewed more to
the right.
```{r}
HSPhy_NextGen_SchoolSum%>%
select(`ERM Diff SD`, `MD Diff SD`)%>%
pivot_longer(c(1:2), names_to = "Practice Cat", values_to = "SD Diff")%>%
ggplot( aes(x=`SD Diff`, color=`Practice Cat`, fill=`Practice Cat`)) +
geom_histogram(alpha=0.6, binwidth = 3) +
scale_fill_viridis(discrete=TRUE) +
scale_color_viridis(discrete=TRUE) +
# theme_ipsum() +
theme(
panel.spacing = unit(0.1, "lines"),
strip.text.x = element_text(size = 8)
) +
labs( y = "",
title = "School Performance Variation by Practice Category",
x = "SD Diff",
caption = "NextGen HS Physics MCAS") +
facet_wrap(~`Practice Cat`) + scale_fill_brewer(palette="Paired") + scale_color_brewer(palette="Paired")
```
## Mathematics and Data vs. Evidence Reasoning and Modeling (Practice Category)
These images seem to suggest that schools with the **highest**
percentage of students classified as `Exceeding or Meeting` expectations on the
MCAS have **lower** levels of variation in performance on
`Mathematics and Data` Items and schools with the **lowest** percentage
of students classified as `Exceeding or Meeting` expectations on the MCAS have the
**highest** levels of variation in performance on
`Mathematics and Data Items`.
```{r}
box_practice_cat_diff<-HSPhy_NextGen_SchoolSum%>%
select(`EM Perf Stat`, `ERM Diff SD`, `MD Diff SD` )%>%
pivot_longer(c(2:3), names_to = "Practice Cat", values_to = "SD Diff")%>%
ggplot( aes(x= `EM Perf Stat`, y=`SD Diff`, fill= `EM Perf Stat`)) +
geom_boxplot() +
scale_fill_viridis(discrete = TRUE, alpha=0.6) +
geom_jitter(color="black", size=0.4, alpha=0.9) +
theme(
plot.title = element_text(size=11),
axis.title.x=element_blank(),
#axis.text.x=element_blank()
) +
labs( y = "SD Diff",
title = "Student Performance Variation by Practice Category",
x = "",
caption = "NextGen HS Physics MCAS") +
facet_wrap(~`Practice Cat`)+scale_fill_brewer(palette="Blues")
box_practice_cat_diff
ggsave("box_practice_cat_diff.png", device="png", width=8, height=6, units="in",dpi=300)
HSPhy_NextGen_SchoolSum%>%
select(`EM Perf Stat`, `ERM Diff SD`, `MD Diff SD` )%>%
pivot_longer(c(2:3), names_to = "Practice Cat", values_to = "SD Diff")%>%
ggplot( aes(x= `Practice Cat`, y=`SD Diff`, fill= `Practice Cat`)) +
geom_boxplot() +
scale_fill_viridis(discrete = TRUE, alpha=0.6) +
geom_jitter(color="black", size=0.4, alpha=0.9) +
#theme_ipsum() +
theme(
plot.title = element_text(size=11),
axis.title.x=element_blank(),
axis.text.x=element_blank()
) +
labs( y = "SD Diff",
title = "Student Practice Cat. Variation by Achievement Level",
x = "",
caption = "NextGen HS Physics MCAS") +
#xlab("")+
facet_wrap(~`EM Perf Stat`)+scale_fill_brewer(palette="Paired")
```
These images seem to suggest that students at all schools seem to have
more difficulty with `Mathematics and Data` items as compared to
`Evidence, Reasoning, and Modeling Items`.
```{r}
HSPhy_NextGen_SchoolSum%>%
select(`EM Perf Stat`, `ERM%`, `MD%` )%>%
pivot_longer(c(2:3), names_to = "Practice Cat", values_to = "%Points")%>%
ggplot( aes(x= `EM Perf Stat`, y=`%Points`, fill= `EM Perf Stat`)) +
geom_boxplot() +
scale_fill_viridis(discrete = TRUE, alpha=0.6) +
geom_jitter(color="black", size=0.4, alpha=0.9) +
#theme_ipsum() +
theme(
plot.title = element_text(size=11)
) +
labs( y = "%Points Earned",
title = "Student Practice Cat. Achievement by Performance Level",
x = "",
caption = "NextGen HS Physics MCAS") +
#xlab("")+
facet_wrap(~`Practice Cat`) +scale_fill_brewer(palette="Blues")
```
```{r}
box_practice_cat_achv<-HSPhy_NextGen_SchoolSum%>%
select(`EM Perf Stat`, `ERM%`, `MD%` )%>%
pivot_longer(c(2:3), names_to = "Practice Cat", values_to = "%Points")%>%
ggplot( aes(x= `Practice Cat`, y=`%Points`, fill= `Practice Cat`)) +
geom_boxplot() +
scale_fill_viridis(discrete = TRUE, alpha=0.6) +
geom_jitter(color="black", size=0.4, alpha=0.9) +
#theme_ipsum() +
theme(
plot.title = element_text(size=11)
) +
labs( y = "%Points Earned",
title = "Student Practice Cat. Achievement by Performance Level",
x = "",
caption = "NextGen HS Physics MCAS") +
#xlab("")+
facet_wrap(~`EM Perf Stat`, scale ="free_y") +scale_fill_brewer(palette="Paired")
box_practice_cat_achv
ggsave("box_practice_cat_achv.png", device="png", width=8, height=6, units="in",dpi=300)
```
## Distribution of School Performance and Variability by Content Reporting Category
Here we can visualize the variability of a school's performance on items
partitioned by Content `Reporting Category` of `Motion and Forces`,
`Energy`, and `Waves` via: `MF%`/`SD MF Diff`, `EN%`/`SD EN Diff`, and
`WA%`/`SD WA Diff`.
```{r}
HSPhy_NextGen_SchoolSum%>%
select(`EM Perf Stat`, `MF Diff SD`, `EN Diff SD`, `WA Diff SD` )%>%
pivot_longer(c(2:4), names_to = "Report Cat", values_to = "SD Diff")%>%
ggplot( aes(x=`SD Diff`, color=`Report Cat`, fill=`Report Cat`, color = "grey")) +
geom_histogram(alpha=0.6, binwidth = 3) +
#scale_fill_viridis(discrete=TRUE) +
#scale_color_viridis(discrete=TRUE) +
#theme_ipsum() +
theme(
panel.spacing = unit(0.1, "lines"),
strip.text.x = element_text(size = 8)
) +
labs( y = "",
title = "School Performance Variation by Content Reporting Category",
x = "SD Diff",
caption = "NextGen HS Physics MCAS") +
facet_wrap(~`Report Cat`) + scale_fill_brewer(palette="Paired") + scale_color_brewer(palette="Paired")
```
```{r}
HSPhy_NextGen_SchoolSum%>%
select(`EM Perf Stat`, `MF%`, `EN%`, `WA%` )%>%
pivot_longer(c(2:4), names_to = "Report Cat", values_to = "% Points")%>%
ggplot( aes(x=`% Points`, color=`Report Cat`, fill=`Report Cat`)) +
geom_histogram(alpha=0.6, binwidth = 3) +
scale_fill_viridis(discrete=TRUE) +
scale_color_viridis(discrete=TRUE) +
#theme_ipsum() +
theme(
panel.spacing = unit(0.1, "lines"),
strip.text.x = element_text(size = 8)
) +
facet_wrap(~`Report Cat`)+
labs( y = "",
title = "Student Performance by Content Reporting Category",
x = "% Points Earned",
caption = "NextGen HS Physics MCAS") + scale_fill_brewer(palette="Paired") + scale_color_brewer(palette="Paired")
#ggtitle("Practice Category Performance")
```
## Motion and Forces vs. Energy vs. Waves (Reporting Category)
These images suggest that schools seem to perform the best nn Motion and Forces items
and the weakest on items in the Waves category. Most schools exhibit similar levels of
variability in performance relative to the state across all reporting
categories. Schools with the ***lowest percentage*** of students
`Exceeding` expectations exhibit ***high variability*** in performance
across all content reporting categories, but seem to have lower
variability on `Waves` items.
```{r}
box_report_cat_achv<-HSPhy_NextGen_SchoolSum%>%
select(`EM Perf Stat`, `MF%`, `EN%`, `WA%` )%>%
pivot_longer(c(2:4), names_to = "Report Cat", values_to = "% Points")%>%
ggplot( aes(x= `Report Cat`, y=`% Points`, fill= `Report Cat`)) +
geom_boxplot() +
scale_fill_discrete(breaks=c('MF%', 'EN%', 'WA%'))+
#scale_fill_viridis(discrete = TRUE, alpha=0.6) +
geom_jitter(color="black", size=0.4, alpha=0.9) +
#theme_ipsum() +
theme(
plot.title = element_text(size=11),
axis.title.x=element_blank(),
axis.text.x=element_blank()
) +
labs( y = "Report Cat%",
title = "School Content Reporting Cat. Performance by Achievement Level",
x = "",
caption = "NextGen HS Physics MCAS") +
#xlab("")+
facet_wrap(~`EM Perf Stat`, scales = "free") + scale_fill_brewer(palette="Blues")+
scale_x_discrete(limit = c("MF%", "EN%", "WA%"))
box_report_cat_achv
ggsave("box_report_cat_achv.png", device="png", width=8, height=6, units="in",dpi=300)
HSPhy_NextGen_SchoolSum%>%
select(`EM Perf Stat`, `MF Diff SD`, `EN Diff SD`, `WA Diff SD` )%>%
pivot_longer(c(2:4), names_to = "Report Cat", values_to = "SD Diff")%>%
ggplot( aes(x= `EM Perf Stat`, y=`SD Diff`, fill= `EM Perf Stat`)) +
geom_boxplot() +
scale_fill_viridis(discrete = TRUE, alpha=0.6) +
geom_jitter(color="black", size=0.4, alpha=0.9) +
theme(
plot.title = element_text(size=11),
axis.title.x=element_blank(),
axis.text.x=element_blank()
) +
labs( y = "SD Diff",
title = "School Performance Variation by Content Reporting Category",
x = "",
caption = "NextGen HS Physics MCAS") +
facet_wrap(~`Report Cat`) + scale_fill_brewer(palette="Blues")
HSPhy_NextGen_SchoolSum%>%
select(`EM Perf Stat`, `MF Diff SD`, `EN Diff SD`, `WA Diff SD` )%>%
pivot_longer(c(2:4), names_to = "Report Cat", values_to = "SD Diff")%>%
ggplot( aes(x= `Report Cat`, y=`SD Diff`, fill= `Report Cat`)) +
geom_boxplot() +
scale_fill_viridis(discrete = TRUE, alpha=0.6) +
geom_jitter(color="black", size=0.4, alpha=0.9) +
#theme_ipsum() +
theme(
plot.title = element_text(size=11),
axis.title.x=element_blank(),
axis.text.x=element_blank()
) +
labs( y = "SD Diff",
title = "School Content Reporting Cat. Variation by Achievement Level",
x = "",
caption = "NextGen HS Physics MCAS") +
#xlab("")+
facet_wrap(~`EM Perf Stat`)+ scale_fill_brewer(palette="Blues")
```
```{r}
HSPhy_NextGen_SchoolSum<-HSPhy_NextGen_SchoolSum%>%
ungroup()
HSPhy_NextGen_SchoolSum<-HSPhy_NextGen_SchoolSum%>%
mutate(`EorM%` = `E%` + `M%`)
#HSPhy_NextGen_SchoolSum
```
## Visualization of School Size, EorM%, MD Diff, ERM Diff
To explore the relationship between the variance in a school's `Diff`
compared to the state on Mathematics and Data items, `MD Diff SD` and a
school's percentage of students meeting or exceeding expectations on the
MCAS, `EorM%`, we also will consider the impact
of `School Size`, `1`: Smallest, `4`: Largest schools, as a control. It appears from our visuals that
`Small` schools have a higher variation in mathematics and data items and
smaller school's typically perform worse on Mathematics and Data items and overall on the
MCAS compared to larger schools.
```{r}
HSPhy_NextGen_SchoolSum%>%
group_by(`School Size`)%>%
summarize(
`Mean EorM%` = mean(`EorM%`),
`Mean MD%` = mean(`MD%`),
`Mean MD Diff SD` = mean(`MD Diff SD`),
`Mean ERM%` = mean(`ERM%`),
`Mean ERM Diff SD` = mean(`ERM Diff SD`)
)
```
```{r}
HSPhy_NextGen_SchoolSum%>%
select(`School Size`, `MD Diff SD`, `ERM Diff SD` )%>%
pivot_longer(c(2:3), names_to = "Practice Cat", values_to = "SD Diff")%>%
ggplot( aes(x= `Practice Cat`, y=`SD Diff`, fill= `Practice Cat`)) +
geom_boxplot() +
scale_fill_viridis(discrete = TRUE, alpha=0.6) +
geom_jitter(color="black", size=0.4, alpha=0.9) +
#theme_ipsum() +
theme(
plot.title = element_text(size=11),
axis.title.x=element_blank(),
axis.text.x=element_blank()
) +
labs( y = "SD Diff",
title = "Student Practice Cat. Variation by School Size",
x = "",
caption = "NextGen HS Physics MCAS") +
#xlab("")+
facet_wrap(~`School Size`, scale = "free") + scale_fill_brewer(palette="Paired")
HSPhy_NextGen_SchoolSum%>%
select(`School Size`, `MD%`, `ERM%` )%>%
pivot_longer(c(2:3), names_to = "Practice Cat", values_to = "%Points")%>%
ggplot( aes(x= `Practice Cat`, y=`%Points`, fill= `Practice Cat`)) +
geom_boxplot() +
scale_fill_viridis(discrete = TRUE, alpha=0.6) +
geom_jitter(color="black", size=0.4, alpha=0.9) +
#theme_ipsum() +
theme(
plot.title = element_text(size=11),
axis.title.x=element_blank(),
axis.text.x=element_blank()
) +
labs( y = "% Points Earned",
title = "Student Practice Cat. Achievement by School Size",
x = "",
caption = "NextGen HS Physics MCAS") +
#xlab("")+
facet_wrap(~`School Size`) + scale_fill_brewer(palette="Paired")
```
However, when you group the schools by `EM Perf Stat`, you find that the
highest performing, `High EM`, `Small` schools have a higher percentage
of students meeting or exceeding expectations.
Across all sizes, it seems that the weakest performing schools have more
variation in mathematics and data and the strongest performing schools
have less variability in `Mathematics and Data` than in
`Evidence, Reasoning, and Modeling`.
```{r}
# Faceted by performance level
HSPhy_NextGen_SchoolSum%>%
group_by(`School Size`, `EM Perf Stat`)%>%
summarize(`Mean EorM%` = mean(`EorM%`),
`Mean MD%` = mean(`MD%`),
`Mean MD Diff SD` = mean(`MD Diff SD`),
`Mean ERM%` = mean(`ERM%`),
`Mean ERM Diff SD` = mean(`ERM Diff SD`)
)
# HSPhy_NextGen_SchoolSum%>%
# select(`School Size`, `EorM%`, `MD Diff SD`, `EM Perf Stat` )%>%
# #pivot_longer(c(2:3), names_to = "Practice Cat", values_to = "SD Diff")%>%
# ggplot( aes(x= `School Size`, y=`EorM%`, fill= `School Size`)) +
# geom_boxplot() +
# scale_fill_viridis(discrete = TRUE, alpha=0.6) +
# geom_jitter(color="black", size=0.4, alpha=0.9) +
# #theme_ipsum() +
# theme(
#
# plot.title = element_text(size=11),
# axis.title.x=element_blank(),
# axis.text.x=element_blank()
# ) +
# labs( y = "SD Diff",
# title = "Students Meeting or Exceeding Expectations . Variation by Achievement Level",
# x = "",
# caption = "NextGen HS Physics MCAS") +
# #xlab("")+
# facet_wrap(~`EM Perf Stat`, scales = "free")
```
# Hypothesis 1: Variation in Mathematics
```{=html}
<style>
div.blue { background-color:#e6f0ff; border-radius: 5px; padding: 20px;}
</style>
```
::: blue
- A school's percentage of students classified as
`Exceeding or Meeting` expectations on the Introductory Physics MCAS
is negatively associated with a school's variance in performance
relative to students in the state on `Mathematics and Data` items,
`SD MD Diff`.
:::
I fit several models with `EorM%` as the response variable and `MD Diff SD` as the key explanatory variable.
I considered additional explanatory variables to control for the following:
Control for the variability in the other largest Science Practice Category, `Evidence, Reasoning, and Modeling`
* Explanatory 2: `ERM Diff SD`
Control for School Size
* Explanatory 3: `School Size`
Control for the variability in the other content Reporting Categories: `Waves`, `Motion and Forces`, and `Energy`
* Explanatory 4: `WA Diff SD`
* Explanatory 5: `MF Diff SD`
* Explanatory 6: `EN Diff SD`
## Fit 1: Explanatory: `MD Diff SD`
When we consider `MD Diff SD` without controlling for other variables, our model associates a $-3.9674$ drop in `EorM%` for a 1 point increase in `MD Diff SD`, $p = 2.73e-09$.
* $p-value: 2.726e-09$
* adjusted $r^2 = .27$
* AIC = 1021.3
* BIC = 1029.5
```{r}
fit_md = lm(`EorM%` ~ (`MD Diff SD`), data = HSPhy_NextGen_SchoolSum)
summary(fit_md)
AIC(fit_md)
BIC(fit_md)
```
## Fit 2: Explanatory: `MD Diff SD` + `ERM Diff SD`
When we control for the variation in the other Science Practice Category, `Evidence Reasoning, and Modeling`,
`ERM Diff SD` is not significant and does not have a significant interaction with `MD Diff SD`.
```{r}
fit_md_erm = lm(`EorM%` ~ (`MD Diff SD` + `ERM Diff SD`), data = HSPhy_NextGen_SchoolSum)
summary(fit_md_erm)
fit_md_erm_interact = lm(`EorM%` ~ (`MD Diff SD` + `ERM Diff SD` + `MD Diff SD` * `ERM Diff SD`), data = HSPhy_NextGen_SchoolSum)
summary(fit_md_erm_interact)
```
## Fit 3: Explanatory: `MD Diff SD` + `WA Diff SD`
When we control for the variation in the Content Reporting Category, `Waves`: `WA Diff SD`, without considering interaction, there is not a statistical significance
to `WA Diff SD`.
However, when we include the interaction between `WA Diff SD` and `MD Diff SD`,
we have that `MD Diff SD` is still statistically significant with an increase in `MD Diff SD` associated with a $-10.02$ decrease in `EorM%`.
There is a statistically significant, $p = 039$ interaction effect between `WA Diff SD` and `MD Diff SD`. For our model we have:
* $p-value: 2.726e-09$
* adjusted $r^2 = .28$
* AIC = 1018
* BIC = 1031
```{r}
fit_md_wa = lm(`EorM%` ~ (`MD Diff SD` + `WA Diff SD`), data = HSPhy_NextGen_SchoolSum)
summary(fit_md_wa)
fit_md_wa_interact= lm(`EorM%` ~ (`MD Diff SD` + `WA Diff SD` + `MD Diff SD`*`WA Diff SD`), data = HSPhy_NextGen_SchoolSum)
summary(fit_md_wa_interact)
AIC(fit_md_wa_interact)
BIC(fit_md_wa_interact)
```
## Fit 3 b): Explanatory: `MD Diff SD`, `WA Diff SD`, `MD Diff SD`*`WA Diff SD`
When we consider adding a second content reporting category, `Motion and Forces`, only `MD Diff SD` is statistically significant. So we will control for content with just one reporting category, `Waves`.
```{r}
fit_md_wa_mf = lm(`EorM%` ~ (`MD Diff SD` + `WA Diff SD` + `MD Diff SD`*`WA Diff SD` + `MF Diff SD` + `MD Diff SD`*`MF Diff SD`), data = HSPhy_NextGen_SchoolSum)
summary(fit_md_wa_mf)
```
## Fit 4: Explanatory: `MD Diff SD` + `WA Diff SD`, `School Size`
When we add the explanatory variable `School Size` to our model, `WA Diff SD` no longer has a statistically significant interaction with `MD Diff SD`. The only statistical significant explanatory variable is `MD Diff SD`.
- A 1 point increase in `MD Diff SD` is associated with a $-9.5141$ decrease in `EorM%`, $p = .0008$
```{r}
fit_md_wa_size = lm(`EorM%` ~ (`MD Diff SD` + `WA Diff SD` + `MD Diff SD`*`WA Diff SD` + `School Size`), data = HSPhy_NextGen_SchoolSum)
summary(fit_md_wa_size)
```
## Fit 5: Explanatory: `MD Diff SD` `School Size`
When we control for the variation in `School Size` and remove content reporting categories from our model, we have that
- A 1 point increase in `MD Diff SD` is associated with a $-3.79$ decrease in `EorM%`, $p = .0008$
- `School Size` is not statistically significant
* $p-value: 1.41e-09$
* adjusted $r^2 = .26$
* AIC = 1023
* BIC = 1034
```{r}
fit_md_size = lm(`EorM%` ~ (`MD Diff SD` + `School Size`), data = HSPhy_NextGen_SchoolSum)
summary(fit_md_size)
AIC(fit_md_size)
BIC(fit_md_size)
```
## Result of Test
Since our original model, `fit_md`, has the ***2nd lowest AIC***, ***lowest BIC***, and ***highest adjusted $r^2$***, and all other models had `MD Diff SD` with negative coefficients and having statistical significance, even when including controls, we can conclude
::: blue
- A school's percentage of students classified as
`Exceeding or Meeting` expectations on the Introductory Physics MCAS
is negatively associated with a school's variance in performance
relative to students in the state on `Mathematics and Data` items,
`SD MD Diff`.
EorM% = -3.67 * MD Diff SD + 40.330
:::
```{r}
stargazer(fit_md, fit_md_wa, fit_md_wa_size, type="text",
dep.var.labels=c("EorM%"),
covariate.labels=c("MD SD Diff", "WA SD Diff", "School Size"), out="models.txt", title = "High School Physics MCAS")
AIC(fit_md_size)
BIC(fit_md_size)
```
## Diagnostic of Final Model, Fit 1: fit_md
When we consider our diagnostics, our model seems to satisfy the assumptions of:
* Linearity: our residuals plot appears to have a few outliers and a slightly negative slope
* Normality of Errors: our standardized residuals seem to fall on the QQ line.
* Homoskedasticity: there is a slight increase in our Scale-Location plot, but not very steep.
Our model does have a few terms that violate Cook's distance. We have 112 and Cook's distance says the 30th, 33rd, and 104th observations are in
violation, since $\frac{4}{n} \approx .04$.
Furthermore, when we remove those observations, and fit a new model, we find `MD Diff SD` continues to be statistically significant with a 1 point increase in `MD Diff SD`: $p =6.62e-12$ associated with $-4.8152$ decrease in `EorM%`.
For our adjusted model we have:
* $p-value: 6.62e-12$
* adjusted $r^2 = .35$
* AIC = 974
* BIC = 982
When we remove the outliers, the direction of our model and significance of our explanatory variables does not change.
```{r}
#HSPhy_NextGen_SchoolSum
plot(fit_md, which = 1:6)
cooks = 4/112
cooks
test<-HSPhy_NextGen_SchoolSum%>%
filter(`School Code` != "00610021")%>%
filter(`School Code` != "00860505")%>%
filter(`School Code` != "08010605")
fit_md_adj = lm(`EorM%` ~ (`MD Diff SD`), data = test)
summary(fit_md_adj)
AIC(fit_md_adj)
BIC(fit_md_adj)
```
```{r}
#plot(fit_md_adj, which = 1:6)
```
## Visuals MD and School Size vs. ERM%
This image displays the association between `MD Diff SD` and `EorM%`. One can see that high variability in `Mathematics and Data` items is associated with lower achievement for schools. Larger schools tend to perform better on the exam compared to smaller schools and have less variation in their mathematics performance although there are some small schools that have very high `EorM%`.
```{r}
scat_md_size<-ggplot(data = HSPhy_NextGen_SchoolSum, aes(x = `MD Diff SD`, y = `EorM%`, color = `School Size`)) +
geom_point() +
geom_smooth(method="lm", se=T)+
labs( y = "EorM%",
title = "Variance in Mathematics vs. Student Achievement",
x = "MD Diff SD",
caption = "NextGen HS Physics MCAS")
scat_md_size
ggsave("scat_md_size.png", device="png", width=8, height=4, units="in",dpi=300)
```
```{r}
# ggplot(data = HSPhy_NextGen_SchoolSum, aes(x = `MD Diff SD`, y = `EorM%`, color = `EM Perf Stat`)) +
# geom_point() +
# geom_smooth(method="lm", se=T)
```
# Hypothesis 2: Reporting Cateogy and School Performance
```{=html}
<style>
div.blue { background-color:#e6f0ff; border-radius: 5px; padding: 20px;}
</style>
```
::: blue
- A school's summary performance on items in a given content
`Reporting Category` as measured by `MF%`, `EN%`, and `WA%`, is
positively associated with the `Reporting Category's` weight within the exam.
:::
## Fit 1: Explanatory:`Report_Cat_Weight`
When we consider `Report_Cat_Weight` without controlling for other variables, our model associates a $.24$ point increase in `Report Cat%` for each 1 point increase in `Report_Cat_Weight`, $p = 6.54e-05$.
* $p-value: 6.537e-05$
* adjusted $r^2 = 0.04382$
* AIC = 2707
* BIC = 2718
```{r}
HSPhy_NextGen_SchoolSum<-HSPhy_NextGen_SchoolSum%>%
mutate(`MF_Weight` = round(100*30/60))%>%
mutate(`EN_Weight` = round(100*18/60))%>%
mutate(`WA_Weight` = round(100*12/60))
#HSPhy_NextGen_SchoolSum
HSPhy_Weight<-HSPhy_NextGen_SchoolSum%>%
select(`MF_Weight`, `EN_Weight`, `WA_Weight`, `MD Diff SD`, `School Size`, `School Name`, `School Code`)%>%
pivot_longer(c(1:3), names_to = "Report Cat", values_to = "Exam Weigt") #%>%
#HSPhy_Weight
HSPhy_Cat<-HSPhy_NextGen_SchoolSum%>%
select(`MF%`, `EN%`, `WA%`, `School Code`, `MD Diff SD`, `School Size`)%>%
pivot_longer(c(1:3), names_to = "Report Cat", values_to = "Report Cat%")%>%
group_by
HSPhy_Cat<-HSPhy_Cat%>%
mutate(Report_Cat_Weight = case_when(
`Report Cat` == "MF%" ~ 50,
`Report Cat` == "EN%" ~ 30,
`Report Cat` == "WA%" ~ 20,
))
#HSPhy_Cat
# HSPhy_Weight<- HSPhy_Weight%>%
# select(`School Name`, `School Code`, `Report Cat`, `Exam Weight`, `% Points`)%>%
# group_by(`School Name`, `School Code`, `Report Cat`)%>%
# summarize(`Exam Weight` = mean(`Exam Weight`),
# `Report Cat %` = mean(`% Points`))%>%
# ungroup()
#
#HSPhy_Cat
fit_weight = lm(`Report Cat%` ~ `Report_Cat_Weight`, data = HSPhy_Cat)
summary(fit_weight)
AIC(fit_weight)
BIC(fit_weight)
```
## Fit 2: `Report_Cat_Weight`, `MD Diff SD`
When we control for the variation in the Science Practice Category, `Mathematics and Data`: `MD Diff SD`,
- An increase in `Report_Cat_Weight`: $1.05e-06$ is associated with a $.24$ increase in `Report Cat%`
- An increase in `MD Diff SD`: $p < 2e-16$ is associated with a $-2.25$ decrease in `Report Cat%`.
For our model we have:
* $p-value: 2.2e-16$
* adjusted $r^2 = 0.37$
* AIC = 2568
* BIC = 2584
```{r}
fit_weight_md = lm(`Report Cat%` ~ `Report_Cat_Weight` + `MD Diff SD` , data = HSPhy_Cat)
summary(fit_weight_md)
AIC(fit_weight_md)
BIC(fit_weight_md)
```
## Fit 2.5: `Report_Cat_Weight`, `MD Diff SD`, with Interaction
When we test for interaction between `Reporting Category Weight`: `Report_Cat_Weight` and `Mathematics and Data`: `MD Diff SD`,
- there is not a significant interaction.
```{r}
fit_weight_md_interact = lm(`Report Cat%` ~ `Report_Cat_Weight` + `MD Diff SD` + `Report_Cat_Weight`*`MD Diff SD`, data = HSPhy_Cat)
summary(fit_weight_md_interact)
#AIC(fit_weight_md_interact)
#BIC(fit_weight_md_interact)
```
## Fit 3: `Report_Cat_Weight`, `MD Diff SD`, `School Size`
When we control for the variation in `MD Diff SD` and `School Size`,
we have that `Report_Cat_Weight`: $p =4.37e-07$, `MD Diff SD`: $p = 0.0318$, and `School Size`: $p = 4.33e-06$ are statistically significant and there is a statistically significant, $p = 4.74e-07$, interaction effect between `MD Diff SD` and `School Size`.
For our model we have:
- an increase in `Report_Cat_Weight` is associated with a $.24$ increase in `Report Cat%`
- E(`Report Cat%`) = 48.29 + 6.83 * `School Size` - .77* `MD Diff SD` - .98(`MD Diff SD` * `School Size`)
* $p-value: 2.2e-16$
* adjusted $r^2 = 0.41$
* AIC = 2546
* BIC = 2569
When we include an interaction between `School Size` and `MD Diff SD`, there is a significant interaction, $p = 4.74e-07$.
```{r}
fit_weight_md_size = lm(`Report Cat%` ~ `Report_Cat_Weight` + `MD Diff SD` + `School Size` + `School Size`*`MD Diff SD`, data = HSPhy_Cat)
summary(fit_weight_md_size)
AIC(fit_weight_md_size)
BIC(fit_weight_md_size)
```
## Result of Test
```{r}
stargazer(fit_weight, fit_weight_md, fit_weight_md_interact, fit_weight_md_size, type="text",
dep.var.labels=c("Report Cat%"))
#covariate.labels=c("Report Cat. Weight", "MD SD Diff", "School Size"), out="report_cat_models.txt", title = "Reporting Category Weight")
```
Since this model has the ***lowest AIC***, ***lowest BIC***, ***highest adjusted $r^2$***, and all terms are statistically significant, we can conclude
::: blue
- A school's summary performance on items in a given content
`Reporting Category` as measured by `Report Cat%`, is
positively associated with the `Reporting Category's` weight within the exam.
`Report_Cat Weight` was statistically significant, even when we control for impact of `MD Diff SD` and`School Size`.
Report Cat% = .239 * Report_Cat_Weight - .77 * MD Diff SD + 6.831 * School Size \n
- .981 * MD Diff SD * School Size + 48.289
:::
It should be noted, that there was a very large increase from $.04$, to $.37$ in $r^2$ once we added `MD Diff SD` as an explanatory variable to our model.
## Diganostic of Fit3: `fit_weight_md_size`
When we consider our diagnostics, our model seems to satisfy the assumptions of:
* Linearity: our residuals plot appears to have a few outliers with the remaining values forming a horizontal band around the 0 line.
* Normality of Errors: Our standardized residual points generally fall along the QQ line.
* Homoskedasticity: there is a slight increase in our Scale-Location plot, but not very steep.
Our model does have some terms that violate Cook's distance. We have 336 observations and Cook's distance says the 141st, 310th, and 311th observations are in
violation, since $\frac{4}{n} \approx .01$.
When we remove those observations, and fit a new model, we find all of our explanatory variables to continue to be statistically significant
For our adjusted model we have:
* $p-value: 2.2e-16$
* adjusted $r^2 = .49$
* AIC = 2437
* BIC = 2460
When we remove the outliers, the direction of our model, significance of our explanatory variables, and signs of their coefficients do not change.
```{r}
#HSPhy_Cat
4/336
plot(fit_weight_md_size, which = 1:6)
test<-HSPhy_Cat%>%
filter(`School Code` != "01520505")%>%
filter(`School Code` != "08010605")
fit_weight_md_size_adj = lm(`Report Cat%` ~ `Report_Cat_Weight` + `MD Diff SD` + `School Size` + `School Size`*`MD Diff SD`, data = test)
summary(fit_weight_md_size_adj)
AIC(fit_weight_md_size_adj)
BIC(fit_weight_md_size_adj)
```
## Visualization of Fit 3: `Report_Cat_Weight`, `MD Diff SD`, `School Size`
```{r}
HSPhy_Cat_Plot<-HSPhy_Cat%>%
mutate(`School Size Fact` = case_when(
`School Size` == 1 ~ "Small",
`School Size` == 2 ~ "Low-Mid",
`School Size` == 3 ~ "Upper-Mid",
`School Size` == 4 ~ "Large"
))%>%
mutate(`School Size Fact` = recode_factor(`School Size Fact`,
"Small" = "Small",
"Low-Mid" = "Low-Mid",
"Upper-Mid" = "Upper-Mid",
"Large" = "Large",
.ordered = TRUE))
#HSPhy_Cat_Plot
mod_plot1<-ggplot(data = HSPhy_Cat_Plot, aes(x = `MD Diff SD`, y = `Report Cat%`, color = `School Size Fact`))+
geom_point() +
geom_smooth(method="lm")+
labs( y = "Report Cat%",
title = "MD SD Diff and School Size vs. Achievement",
x = "MD Diff SD",
caption = "NextGen HS Physics MCAS")+scale_color_brewer(palette="Paired")
mod_plot1
ggsave("scat_md_weight_size_interact_achv.png", device="png", width=8, height=6, units="in",dpi=300)
#mod_plot2<- mod_plot1 + facet_wrap(~`Report_Cat_Weight`, scales = "free")
#mod_plot2
```
```{r}
scat_md_weight<-ggplot(data = HSPhy_Cat, aes(x = `MD Diff SD`, y = `Report Cat%`, color = `Report_Cat_Weight`)) +
geom_point() +
geom_smooth(method="lm", se=T)+
labs( y = "Report Cat%",
title = "Variance in Mathematics and Category Weight vs. Achievement",
x = "MD Diff SD",
caption = "NextGen HS Physics MCAS")#+
#facet_wrap(~`Report_Cat_Weight`, scales = "free")
scat_md_weight
ggsave("scat_md_weight.png", device="png", width=8, height=4, units="in",dpi=300)
```
```{r}
# ggplot(data = HSPhy_Cat, aes(x = `MD Diff SD`, y = `Report Cat%`, color = `School Size`)) +
# geom_point() +
# geom_smooth(method="lm", se=T)+
# labs( y = "Report Cat%",
# title = "Variance in Mathematics and School Size vs. Achievement",
# x = "MD Diff SD",
# caption = "NextGen HS Physics MCAS")+
# facet_wrap(~`Report_Cat_Weight`, scales = "free")
```
```{r}
# ggplot(data = HSPhy_Cat, aes(x = `MD Diff SD`, y = `Report Cat%`, color = `Report_Cat_Weight`)) +
# geom_point() +
# geom_smooth(method="lm", se=T)+
# labs( y = "Report Cat%",
# title = "Variance in Mathematics and Category Weight vs. Achievement",
# x = "MD Diff SD",
# caption = "NextGen HS Physics MCAS")+
# facet_wrap(~`Report_Cat_Weight`, scales = "free")
```
```{r}
# ggplot(data = HSPhy_Cat, aes(x = `MD Diff SD`, y = `Report Cat%`, color = `School Size`)) +
# geom_point() +
# geom_smooth(method="lm")+
# labs( y = "Report Cat%",
# title = "MD SD Diff, Category Weight, and School Size vs. Achievement",
# x = "MD Diff SD",
# caption = "NextGen HS Physics MCAS")+
# facet_wrap(~`Report_Cat_Weight`, scales = "free")
```
```{r}
#mod_plot<-ggplot(data = HSPhy_Cat, aes(x = `MD Diff SD`, y = `Report Cat%`, color = `School Size`))
#mod_plot + geom_point() + geom_smooth(method = lm,se = FALSE, fullrange = TRUE) #+ #scale_color_brewer(type = "qual", palette = 3)
# geom_point() +
# geom_smooth(method="lm")+
# labs( y = "Report Cat%",
# title = "MD SD Diff, Category Weight, and School Size vs. Achievement",
# x = "MD Diff SD",
# caption = "NextGen HS Physics MCAS")+
# facet_wrap(~`Report_Cat_Weight`, scales = "free")
```
# Conclusion
When considering the performance of Massachusetts students on the Next Generation High School Introductory Physics Exam, we find that:
1. Student performance as measured by `Reporting Cat%` and `EorM%` were negatively associated with increased variability in `Mathematics and Data`, `MD Diff SD`.
This suggests that schools could benefit by spending time reviewing and supporting science teachers in implementing evidence based, best practices for the teaching of Mathematics within the Physics classroom.
2. Student performance as measured by `Reporting Cat%` was positively associated with the `Report_Cat_Weight`.
There was no significant interaction between `Reporting Cat%` and `MD Diff SD`. This suggests that schools could benefit by creating a scope and sequence for the calendar year that ensures that all content areas from the Massachusetts Curriculum framework are addressed within the academic year. Perhaps items in the `Waves` category are not receiving enough instructional time within the school year as compared to those in the highest weighted `Motion and Forces` category.
One could examine the G8 Science exam as well as the G5 Science exam by `Science Practice Category` and `Content Reporting Category` to see if there are similar trends exhibited in earlier grade levels.
# References